Introduction to the Java Robot Class

Baldwin shows you how to write a Java program that will programmatically invoke mouse clicks on non-Java programs, and programmatically enter text into non-Java programs.

Note:  The original published version of this lesson had a bug in the program named Robot05.  In particular, the reference variable named keyInput[] should have been declared static.

Published:  May 20, 2003
By Richard G. Baldwin

Java Programming Notes # 1472

Preface

Programming in Java doesn't have to be dull and boring.  In fact, it's possible to have a lot of fun while programming in Java.  This is the first lesson in a miniseries that show you how to use the Robot class to write programs that are both fun and useful.

New features in SDK Version 1.3

One of the new features that was released in SDK Version 1.3 was the Robot class.  According to Sun,

"This class is used to generate native system input events for the purposes of test automation, self-running demos, and other applications where control of the mouse and keyboard is needed. The primary purpose of Robot is to facilitate automated testing of Java platform implementations."

Different from the system event queue

I have published several tutorial lessons that show you how to create events and post them to the system event queue.  (For example, see lesson 104 at http://www.dickbaldwin.com/tocmed.htm.)  However, it is important to note that using a Robot object to fire events is significantly different from posting events to the system event queue.

Here is how Sun compares the Robot class and the system event queue.

"Using the class to generate input events differs from posting events to the AWT event queue or AWT components in that the events are generated in the platform's native input queue. For example, Robot.mouseMove will actually move the mouse cursor instead of just generating mouse move events."

Potential problem areas

Basically, a Robot object makes it possible for your program to temporarily take over control of the mouse and the keyboard.  However, this may not be possible on all operating systems.  For example, Sun has this to say.

"Note that some platforms require special privileges or extensions to access low-level input control. If the current platform configuration does not allow input control, an AWTException will be thrown when trying to construct Robot objects."

Sun goes on to provide some examples of cases where an AWTException might be thrown.

A word of caution

A runaway Java Robot object has the ability to wrest control away from the human user, so you need to be a little careful.  For example, if you allow your Java Robot program to go into an infinite loop, making mouse moves, clicking the mouse, and entering keystrokes, you may find that the only practical way to regain control of your computer is to either turn off the power or press the reset button to force your computer to restart.

Three lessons in the miniseries

According to my current planning, this miniseries on the Robot class will consist of three lessons.  This first lesson will demonstrate the low-level nature of the behavior of an object of the Robot class.  This will be accomplished by showing you how to create a Java robot that can manipulate other non-Java programs, such as Windows Notepad and Internet Explorer.

The second lesson in the miniseries will show you how to use a robot to perform automatic testing on a Java GUI.

The third lesson will show you how to write an animated robot program to provide a visual demonstration of the use of a Java GUI.

Viewing tip

You may find it useful to open another copy of this lesson in a separate browser window.  That will make it easier for you to scroll back and forth among the different listings and figures while you are reading about them.

Supplementary material

I recommend that you also study the other lessons in my extensive collection of online Java tutorials.  You will find those lessons published at Gamelan.com.  However, as of the date of this writing, Gamelan doesn't maintain a consolidated index of my Java tutorial lessons, and sometimes they are difficult to locate there.  You will find a consolidated index at www.DickBaldwin.com.

Preview

I will discuss two sample programs named Robot04 and Robot05 in this lesson.  The sample programs will illustrate:

Discussion and Sample Code

I will begin with a discussion of the program named Robot04, followed by a discussion of the program named Robot05.

Description of the program named Robot04

The Robot class makes it possible for your Java program to temporarily take control of the mouse and keyboard input functionality at the operating-system level.

(In fact, if you accidentally allow your program to take control of the mouse and the keyboard in an infinite loop, the program can be very difficult to terminate.)

This program illustrates the low-level nature of an object of the Robot class.  In this program, a Robot object is designed to click the Close button on a non-Java program (such as Internet Explorer) causing that program to terminate.

Should work in Windows 1024 by 768 mode

This program should run successfully without modification if you are running Windows with a screen resolution of 1024 by 768 pixels.  If you are running under a different operating system, or a different screen resolution, you may need to modify the coordinate values in the program to cause this program to behave as described.  (A comment is provided in the code to show you where to make the modification.)

Open Internet Explorer in full-screen mode

To see the intended behavior of this program, start Internet Explorer, (or some other program with a Close button in its upper right corner) and maximize it to full-screen mode.  This should cause the Close button with the X for that program to appear in the upper right-hand corner of the screen.

This program will click the Close button

Then run this program at the command prompt.  There is no GUI for this program.  The program simply starts running, performs the programmed action, and terminates.

If all goes well, this program will use an object of the Robot class to position the mouse pointer over the Close button belonging to the other program.  Then this program will use the Robot object to click that button.  The result of clicking the button will be the same as if a human user had clicked the Close button on that program.

(Frequently, the result of clicking the Close button on a program's main GUI is to terminate the program.  That is the case with Internet Explorer.  However, other programs, such as Windows Notepad, may bring up a dialog asking the user to confirm something such as saving a file.  This program is not designed to deal with such a dialog.)

If things don't work as described ...

If you don't get this result, try adjusting the coordinate values in the program to make certain that the mouse pointer is automatically positioned on the Close button before the mouse click is executed.

(Delays are inserted in the behavior of this program so that you can view the process in slow motion.)

Requires version 1.3 or later

The Robot class was first released in SDK Version 1.3, so you will need to be running Version 1.3 or later.  This program was tested on my machine using SDK 1.4.1 under WinXP

The main method for the program named Robot04

I will discuss this program in fragments.  A complete listing of the program is shown in Listing 10 near the end of the lesson.

The program begins in Listing 1 where the main method starts by instantiating a new object of the Robot class.
  
public class Robot04{
  public static void main(String[] args)
                             throws AWTException{
    Robot robot = new Robot();

Listing 1

Mouse and keyboard input under program control

As mentioned earlier, an object of the Robot class makes it possible for your program to temporarily gain control of the mouse and keyboard.

(The Robot class also provides some other interesting features, such as the ability to create an image containing pixels read from the screen, which are not discussed in this lesson.  See Sun's documentation for a description of those features.)

Several instance methods are available

The Robot class provides several instance methods, (including the following), by which your program can produce mouse and keyboard input, just as though that input were being provided by a human user.

Move the mouse pointer

The code in Listing 2 causes the mouse pointer to move to a location 1005 pixels to the right and 10 pixels down from the upper left-hand corner of the screen.

(If you watch carefully while the program is running, you will see the mouse pointer move to this location.

If your program attempts to move the mouse pointer to a location specified by negative coordinates, it will move to the 0,0 location in the upper-left corner of the screen.  If your program moves the mouse pointer off the screen with positive coordinates, the pointer will simply disappear.)

    robot.mouseMove(1005,10);

Listing 2

Should point to the Close button of Internet Explorer

For a screen resolution of 1024 by 768 with a Windows look and feel, the coordinates 1005, 10 should cause the mouse pointer to point to the Close button on a full-screen version of Internet Explorer.  If this is not the case on your system, you may need to modify these coordinate values to cause the mouse to point to the Close button.

Insert a delay

The code in Listing 3 inserts a delay in the process by causing the thread to sleep for two seconds (2000 milliseconds).  This should make it possible for you to view the pointer in its new location before anything else happens.

    robot.delay(2000);

Listing 3

What about InterruptedExceptions?

Sometimes when you put a thread to sleep, you may want to be able to catch InterruptedExceptions that may occur on that thread.  If you have that need, you should use the sleep method of the Thread class instead of the delay method of the Robot class.

Press the left mouse button and delay again

The code in Listing 4 invokes the mousePress method on the Robot object to execute a press of the left mouse button at the current position of the mouse pointer.
  
    robot.mousePress(InputEvent.BUTTON1_MASK);
    robot.delay(2000);

Listing 4

The mousePress method accepts an incoming int parameter that specifies which mouse button is pressed.  Suitable values for specifying the left, center, and right mouse buttons are found as constants in the InputEvent class as shown below:

(Note that the Robot class was updated in SDK V1.4 to support a mouse wheel.  According to Sun, the mouseWheel method of the Robot class "Rotates the scroll wheel on wheel-equipped mice."  That capability is not illustrated in this lesson.)

Does anything interesting happen?

Whether or not something interesting happens as a result of invoking the mousePress method in Listing 4 depends on what the mouse pointer is pointing to when the method is invoked.  The same thing will happen that would happen if a human user were to press the left mouse button when the mouse pointer is pointing to the same location on the screen.

If the pointer happens to be pointing at a sensitive location on the screen, something interesting will happen.  If the mouse pointer happens to be pointing to a non-sensitive location on the screen, the mouse press will simply be ignored and nothing interesting will happen.

(As an interesting exercise, try modifying this program to invoke a right-button click (BUTTON3_MASK) on a blank area of a Windows screen.  This should cause a typical Windows popup menu to appear.)

Another delay

The code in Listing 4 includes another two-second delay.  If the mouse pointer is pointing to the Close button when the mousePress method is invoked, you should be able to see the Close button in its depressed state for two seconds. 

Does not terminate Internet Explorer

Simply pressing the left mouse button on the Close button on Internet Explorer won't terminate the program.  A mouse press followed by a mouse release is required to terminate the program.

Release the left mouse button

The code in Listing 5 invokes the mouseRelease method causing the program to behave as if a human user has released the left mouse button (after first pressing that button).
  
    robot.mouseRelease(InputEvent.BUTTON1_MASK);
  }//end main

}//end class Robot04

Listing 5

Assuming that the mouse pointer is pointing to the Close button on Internet Explorer, the code in Listing 5 should cause the Close button to pop back out and should cause the Internet Explorer program to terminate.  (However, it will probably happen so fast that you probably won't see the button pop back out.)

The code in Listing 5 also ends the main method, ends the class definition, and thus ends the program.

Recap for the program named Robot04

The Java program named Robot04 causes the mouse to move to a location that is believed to be the location of the Close button for a running instance of Internet Explorer.

(In the next lesson, I will show you how to determine the location of components in a Java GUI with more certainty.)

Having moved the mouse pointer to that location, Robot04 clicks the left mouse button in an attempt to cause the running instance of Internet Explorer to terminate.  If the location of the mouse pointer was correct, Internet Explorer will terminate.

Now let's move along to another program that demonstrates use of the keyboard by a Robot object.

Description of the program named Robot05

In addition to creating low-level mouse inputs, an object of the Robot class can create low-level keyboard inputs, just as though a human user were pressing the keys on the keyboard.

Designed to work specifically with Windows OS

As written, this program will run correctly only under the Windows operating system.  This is because it specifically makes use of the Windows program named Notepad.

(However, the program could easily be modified to use a similar editor program under a different operating system.)

Low-level keyboard input

This program illustrates the low-level nature of the behavior of an object of the Robot class by:

Running the program

This Java program does not have a GUI.  To see the intended behavior of this program, simply start the program running from the command line under the Windows operating system.

Notepad program will start

When this Java program starts running, it causes the Windows Notepad program to start running in a separate process with a new empty document.

Enter text into the Notepad document and terminate

Then this program uses a Robot object to enter the word Hello into the Notepad document.  After that, the Java program terminates.

When the Java program terminates, the Notepad program is still running, and can be saved to a file, or terminated by the user.

Requires SDK V1.3 or later

As mentioned earlier, the Robot class was first released in SDK Version 1.3.  Therefore, as was the case in the previous program, you will need to be running Version 1.3 or later to successfully compile and execute this program.  This program was tested on my machine using SDK 1.4.1 under WinXP

Will discuss in fragments

As usual, I will discuss this program in fragments.  You can view a complete listing of Robot05 in Listing 11 near the end of the lesson.

Need some keycode data

The purpose of this program is to demonstrate the ability of a Java program to enter character data into a non-Java program.  Therefore, we will need some keycode data.

The code in Listing 6 creates an array object containing the virtual keycodes for the word hello.  This data will be used later with the Robot.keyPress and Robot.keyRelease methods, and the KeyEvent.VK_SHIFT virtual keycode, to mimic the behavior of a human user entering the characters Hello into a Notepad document.
  
public class Robot05{
  //Create an array of keycode data
  static int keyInput[] = {
      KeyEvent.VK_H,
      KeyEvent.VK_E,
      KeyEvent.VK_L,
      KeyEvent.VK_L,
      KeyEvent.VK_O
  };//end keyInput array

Listing 6

Virtual keycodes are used

In one of the technical documents on its site, Sun has this to say about specifying keycodes for use with a Robot object:

"Notice that virtual keycodes are used to enter keystrokes; keycodes are not the same as Java characters. KeyEvent.VK_A corresponds to pressing the unshifted 'A' key on a keyboard. If you specify 'A' or 'a' instead of KeyEvent.VK_A, you get unexpected results."

Start Notepad running

The main method begins in Listing 7.  The code in Listing 7 starts the Notepad program running in a separate process.  This should cause the Notepad program to start running and to become the active window, capable of accepting input from the keyboard.

  public static void main(String[] args)
                 throws AWTException,IOException{

    Runtime.getRuntime().exec("notepad");

Listing 7

Runtime is not affiliated with the Robot class

Note that the exec method used in Listing 7 has nothing to do with the Robot class.  The Runtime class and its methods have been available since JDK1.0.

Here is what Sun has to say about the Runtime class.

"Every Java application has a single instance of class Runtime that allows the application to interface with the environment in which the application is running. The current runtime can be obtained from the getRuntime method."

Methods of the Runtime class

The Runtime class provides about twenty instance methods that allow you do unrelated things such as:

The exec method of the Runtime class

In addition, the exec method of the Runtime class makes it possible to "execute a specified string as a command in a separate process."

The code in Listing 7 executes the string "notepad" as a command in a separate process.  This, in turn, causes the Windows program named Notepad to start running and to become the active window.

Get a Robot object

The code in Listing 8 gets and saves a reference to a new Robot object, which will be used to enter text characters into the Notepad edit window.
  
    Robot robot = new Robot();

Listing 8

Enter the characters into the Notepad edit window

The code in Listing 9 extracts the keycodes previously stored in the array and invokes the keyPress method to mimic the action of a human typist.  This causes the characters represented by those keycodes to be entered into the Notepad editor.
  
    robot.keyPress(KeyEvent.VK_SHIFT);
    for (int cnt2 = 0;
                 cnt2 < keyInput.length; cnt2++){
      if(cnt2 > 0){
             robot.keyRelease(KeyEvent.VK_SHIFT);
      }//end if
      robot.keyPress(keyInput[cnt2]);
      //Insert a one-half second delay between
      // characters.
      robot.delay(500);
    }//end for loop

  }//main
}//end class Robot05

Listing 9

Note upper and lower case characters

Note the combined use of the keyPress and keyRelease methods along with KeyEvent.VK_SHIFT virtual keycode to mimic the user holding down the Shift key for entry of the first character, H.

A one-half second delay

The code in Listing 9 also inserts a one-half second delay following the entry of each character so that you can see them being individually entered into the Notepad editor.

End of the program

The code in Listing 9 also signals the end of the main method and the end of the program.

After all of the characters have been entered into the Notepad editor, the Java program terminates.

Recap for program Robot05

The program named Robot05 starts a new instance of the program named Notepad running in a separate process, and then enters the characters Hello into the Notepad document.  Then the Java program terminates, leaving the Notepad program running.

Run the Program

I encourage you to copy the code from Listings 10 and 11 into your text editor, compile it, and execute it.  Experiment with it, making changes, and observing the results of your changes.

Remember, however, that you must be running Java V1.3 or later to compile and execute these programs, because they make use of Java features that were first released in V1.3.  (In case you decide to make use of the mouseWheel method of he Robot class, you must be running SDK V1.4 or later.)

Summary

In this lesson, I have taught you:

What's Next?

The next lesson in this miniseries will show you how to use a robot to perform automatic testing on a Java GUI.

The third lesson will show you how to write an animated robot program to provide a visual demonstration of the use of a Java GUI.

Complete Program Listings

Complete listings of the programs discussed in this lesson are shown in Listings 10 and 11 below.
  
/*File Robot04.java
Copyright 2003 R.G.Baldwin

This program illustrates the low-level nature of
the behavior of an object of the Robot class by
programming a Robot object to click the Close
button of a non-Java program (such as Internet
Explorer) and causing that program to terminate.

This program should work without modification if
you are running WinXP with 1024 by 768 screen
resolution.  If you are running under a different
OS, or a different screen resolution, you may
need to modify the coordinate values in the
program.

To see the desired behavior of this program,
start Internet Explorer, (or some other program)
and maximize it to full-screen mode.  This should
cause the Close button with the X for that
program to appear in the upper right-hand corner
of the screen.

Then run this program.  If all goes well, this
program will use an object of the Robot class
to position the mouse pointer over the Close
button of the other program and then click that
button.  The result will be the standard result
of clicking the Close button on that program.
Frequently, the standard result of clicking the
Close button is to terminate the program.

If you don't get this result, try adjusting the
coordinate values in the program to make certain
that the mouse pointer is automatically
positioned on the Close button before the mouse
click is executed.  Delays are inserted so that
you can view the process in slow motion.

Tested using SDK 1.4.1 under WinXP
************************************************/

import java.awt.*;
import java.awt.event.*;

public class Robot04{
  public static void main(String[] args)
                             throws AWTException{
    Robot robot = new Robot();
    //You may need to modify these coordinates if
    // they fail to place the mouse pointer on
    // the Close button of the non-Java program
    // that you are trying to terminate.
    robot.mouseMove(1005,10);

    //Press and then release the left mouse
    // button.  The delays are provided so that
    // you can view the action.  Watch the upper-
    // right corner of the screen when you run
    // this program.
    robot.delay(2000);
    robot.mousePress(InputEvent.BUTTON1_MASK);
    robot.delay(2000);
    robot.mouseRelease(InputEvent.BUTTON1_MASK);
  }//end main
  //-------------------------------------------//

}//end class Robot04

Listing 10

 

/*File Robot05.java
Copyright 2003 R.G.Baldwin

NOTE:  THE ORIGINAL VERSION AS ORIGINALLY
PUBLISHED HAD A BUG.  SPECIFICALLY, I FAILED
TO DECLARE keyInput[] AS STATIC AND THEREFORE
IT CANNOT BE ACCESSED FROM WITHIN main.

Note:  As written, this program will only run
correctly under the Windows operating system,
because it makes use of the Windows program named
Notepad.  However, it could easily be modified to
use a similar program under a different operating
system.

This program illustrates the low-level nature of
the behavior of an object of the Robot class by:

1. Starting the Windows Notepad editor program
in a separate process.
2. Programming a Robot object to enter text into
the Notepad document.

To see the desired behavior of this program,
simply start the program running under the Windws
operating system.

When the program starts running, it causes the
Windows Notepad program to start running with
a new empty document.

Then this program uses a Robot object to enter
the word Hello into the Notepad document, and
terminates.

At this point, the Notepad program is still
running, and can be saved to a file, or
terminated by the user.

Tested using SDK 1.4.1 under WinXP
************************************************/

import java.awt.*;
import java.awt.event.*;
import java.io.IOException;

public class Robot05{
  //Create an array of keycode data
  static int keyInput[] = {
      KeyEvent.VK_H,
      KeyEvent.VK_E,
      KeyEvent.VK_L,
      KeyEvent.VK_L,
      KeyEvent.VK_O
  };//end keyInput array

  public static void main(String[] args)
                 throws AWTException,IOException{

    //Start the Windows Notepad program running
    // in a separate process.  It should become
    // the active window, capable of accepting
    // input from the keyboard.
    Runtime.getRuntime().exec("notepad");

    //Get a Robot object that will be used to
    // enter characters into the Notepad document
    Robot robot = new Robot();

    //Enter the keycodes contained in the
    // keyInput array into the Notepad document.
    // Make the first character upper case and
    // the remaining characters lower case.
    robot.keyPress(KeyEvent.VK_SHIFT);
    for (int cnt2 = 0;
                 cnt2 < keyInput.length; cnt2++){
      if(cnt2 > 0){
             robot.keyRelease(KeyEvent.VK_SHIFT);
      }//end if
      robot.keyPress(keyInput[cnt2]);
      //Insert a one-half second delay between
      // characters.
      robot.delay(500);
    }//end for loop
  }//main
}//end class Robot05

Listing 11

Copyright 2003, Richard G. Baldwin.  Reproduction in whole or in part in any form or medium without express written permission from Richard Baldwin is prohibited.

About the author

Richard Baldwin is a college professor (at Austin Community College in Austin, TX) and private consultant whose primary focus is a combination of Java, C#, and XML. In addition to the many platform and/or language independent benefits of Java and C# applications, he believes that a combination of Java, C#, and XML will become the primary driving force in the delivery of structured information on the Web.

Richard has participated in numerous consulting projects, and he frequently provides onsite training at the high-tech companies located in and around Austin, Texas.  He is the author of Baldwin's Programming Tutorials, which has gained a worldwide following among experienced and aspiring programmers. He has also published articles in JavaPro magazine.

Richard holds an MSEE degree from Southern Methodist University and has many years of experience in the application of computer technology to real-world problems.

Baldwin@DickBaldwin.com

-end-