Using Robot to automize actions in non-java applications

[JeroenV]

Hello,

I've recently been trying to utilize the java.awt.Robot to automize certain mouse/keyboard functions without using the actual mouse. I basically plan to use a fake mouse object like here:

       
 MouseAdapter mouseImpl = new MouseAdapter() {

            private Component lastPressed;

            @Override
            public void mousePressed(MouseEvent e) {
                redirectMouseEvent(e);
            }

            @Override
            public void mouseReleased(MouseEvent e) {
                redirectMouseEvent(e);
            }

            @Override
            public void mouseClicked(MouseEvent e) {
                redirectMouseEvent(e);
            }

            @Override
            public void mouseMoved(MouseEvent e) {
                redirectMouseEvent(e);
            }

            @Override
            public void mouseDragged(MouseEvent e) {
                redirectMouseEvent(e);
            }

            private void redirectMouseEvent(MouseEvent e) {
                Component redirectTo = SwingUtilities.getDeepestComponentAt(panel, e.getX(), e.getY());
                if (e.getID() == MouseEvent.MOUSE_PRESSED) {
                    lastPressed = redirectTo;
                } else if (e.getID() == MouseEvent.MOUSE_DRAGGED || e.getID() == MouseEvent.MOUSE_RELEASED) {
                    redirectTo = lastPressed;
                }
                if (redirectTo != null) {
                    lastMouseX = e.getX();
                    lastMouseY = e.getY();
                    panel.repaint(); //this line is just to update the glass pane
                    e = SwingUtilities.convertMouseEvent(panel, e, redirectTo);
                    java.awt.Toolkit.getDefaultToolkit().getSystemEventQueue().postEvent(e);
                }
            }
        };
        remoteFrame.getContentPane().addMouseListener(mouseImpl);
        remoteFrame.getContentPane().addMouseMotionListener(mouseImpl);

However the program I'm trying to do this in isn't in any way java related. My question is, can I still use a fake mouse on this program (perhaps if I can open the program inside my own client) or is this impossible. (if so.. any suggestions?)



Thanks in advance,
Jeroen V.

[kjkrum]

Yes, you can. I once wrote a small Java program to "press" a series of keys at set intervals to twist chants for my brother's paladin in Dark Age of Camelot.

[JosAH]

It can't be done; an indication is in your own redirectMouseEvent( ... ) method: it needs a target Component in the same JVM; if the other application isn't even a Java application there are no Components ...

kind regards,

Jos

[kjkrum]

Hm... well, maybe not like that. But you can use Robot#mouseMove(...), #mousePress(...), etc. as well as #keyPress(...), etc. to interact with non-Java programs.

[JosAH]

Hm... well, maybe not like that. But you can use Robot#mouseMove(...), #mousePress(...), etc. as well as #keyPress(...), etc. to interact with non-Java programs.

Sure, but then the Robot shoots with its eyes closed, i.e. it doesn't know where the event lands.

kind regards,

Jos

[DarrylBurke]

You can use Robot to first take a screenshot, analyze that image to locate a known graphic, then you can compute the correct coordinates for the Robot to initiate mouse interactions.

I once wrote a Windows Minesweeper solver which located the smiley face on the reset button for reference coordinates (It could solve the Advanced level in one or two seconds). And then Microsoft totally changed the Minesweeper UI in Vista and my program became useless

db

[JeroenV]

So, you're saying that it's impossible to run a non-java program inside my client to read its output. As another question, is it possible to fake mouse/keyboard that certain program when it's minimized?

[DarrylBurke]

So, you're saying that it's impossible to run a non-java program inside my client

Correct.

is it possible to fake mouse/keyboard that certain program when it's minimized?

You need to be much more specific before anyone can answer that meaningfully.

db

[JeroenV]

Correct.


You need to be much more specific before anyone can answer that meaningfully.

db

Well, with that I mean that the program would create a mouse/keyboard object inside that program that does certain operations while I could continue doing other things with my own mouse.

For example, I would have a bot that goes through my mail using a fake mouse while I'm checking these forums with the real mouse. The catch here is that I would want the bot to keep going through the mail even though I minized the mail program. So I can't just let the fake mouse use normal screen coordinates, I would need to input and output coordinates in the mail viewing program. Which is why I think I would need to open the program inside my botclient, so my botclient can keep track of those coordinates.