so im working on an independent project, and I'm wondering how capable java is.

I want my program to be able to automate mouse clicks on a non-java based program. I found the robot class, which can be used to move and click the mouse; however is there a way to detect button-sensitive areas of the screen? Of course, there are ways to implement the program without being able to detect these sensitive areas, but it could be helpful

also, perhaps more importantly, is there a way to grab text from a screen and tokenize it. There will be a box with text placed into it by the non-java program, and I need to grab this text. The box is non-editable though. I'm gonna keep looking myself, but if anyone is already familiar with a given java class, a lead could make it a little easier for me.

btw this isn't for school, so don't worry about that

12/22/2005 2:35:36 PM

if you're looking for a way to do automated software testing there are tools specifically for that, like Rational Robot.
of course that's about $10k for a single license.....

12/22/2005 2:44:36 PM

Ha sounds like you are trying to get a program that will help you cheat those only pay-surf bar things.

12/22/2005 2:54:09 PM

nah, i don't really want to automate testing, and i don't even know what you are talking about when you say pay-surf bar, but I really don't want to go into the specifics of what I'm trying to do

if anyone could help point me in the right direction, with a specific class object or method that would be really helpful though

to clarify, the program has a gui, on the bottom portion there is a scrollable read-only text box, i need to grab the text from this box and tokenize it, i mean i can tokenize using readfile and all that, but not sure how i can grab the text from this read-only box, i guess that is my main obstacle right now

12/22/2005 3:10:09 PM

The only way to "detect" a button or other GUI element in the host OS would be to use JNI to connect to the host OS's GUI event dispatcher.


You'd be better off using a native library for the OS in question.

12/22/2005 3:13:52 PM

the os is gonna be winXP, i have actually thought of a few ways around having to detect buttons, namely just using absolute coordinates, so i guess that isn't as much of a concern at the moment, especially since I'd like to be able to use java since that is the lang that i am the freshest with at the moment

but when you say native library, you are referring to assembly, or does every os have a higher level than assembly that is native to it? sort of a quasi-assembly if that makes any sense, i'm not a csc major so when things get down to the os level i get all and stuff

12/22/2005 3:20:16 PM

he's fully automating rainbow table generation HAXOR HAXOR! :p

12/22/2005 3:28:20 PM

windows has a number of API's that can be used to find, enumerate, and identify windows, buttons, etc. and to return their various properties (including text). I would assume you can call these from JAVA

look into the

FindWindow (locates a window based on its class or title bar text)
FindWindowEx (locates a control within a window)
GetWindowText (returns the text property of most windows and controls)

GetWindowRect (returns a RECT structure corresponding to the edges of a window or control)

GetCursorPos (gets the cursor position)
SetCursorPos (sets the cursor position)

SendMessage (can be used to send WM_CLICK, WM_LBUTTONDOWN, WM_LBUTTONUP, WM_MOUSEMOVE even messages without actually moving the cursor so normal computer use isn't interrupted)

APIs

[Edited on December 22, 2005 at 3:41 PM. Reason : ]

12/22/2005 3:38:22 PM

^ thats what i mean by library.

And you should be able to call them thru JNI, though it can be a pain in the ass. Google around for something similar to what you're trying to do.

Although you'd be better off just using VB.

If your concern here is learning another language, you might look into j#.

12/22/2005 3:43:14 PM

^^sweet, thanks man that looks really helpful, i'll look into myself, but maybe you could briefly explain the idea of a windows api, is that kinda like command prompt stuff?

i'm not too familiar with coding at the os level, i'm way more familiar with coding at the higher-language level, ie java, c/c++ etc....can windows scripts or programs (whatever they would be called) be embedded inside java code?

can someone just give me a brief summary of how someone would program in windows

^yea, i mean free time is sort of tight, so i'd like to not have to learn another language if at all possible, what is jsharp? obviously a sub/super set of java, but does it provide more direct control to os libraries or something?

and yea, i'm definetly gonna google all this, that's how i found the robot class, i just wanted to put the feelers out to have a better idea of what to google and stuff, you all have been very helpful, keep it coming if you can

[Edited on December 22, 2005 at 3:49 PM. Reason : asdf]

12/22/2005 3:46:01 PM

An API is a set of commands that are stored in a library that allow you to access data and functions of someone elses code.

For example, this is the API documentation for Java. It explains all the functions and classes within java and how to use them.

This is the Massive Windows API refrence from microsoft.


What the Windows API does is allows you to access certain data and functions within windows. Such as getting the current cursor position, finding the attributes of a window, creating new windows, etc.. etc... There is a ton of stuff there and if you dont have any previous experience with the Windows API or JNI you're going to need to do alot of learning. Its not neccisarily hard, it will just take time to familiarize yourself with how it all works. This is where googling some tutorials might come in handy.

Of course you might just want to look to see if someone has already create a java bot for controlling windows. Or to see if someone has already made a JNI bridge into the windows api (very likely).

12/22/2005 3:53:27 PM

ok, i get that, but how would the windows functions execute, through a command prompt, or can i just embed it inside java code (seems unlikely, since the java compiler probably won't recognize the functions) so how would i run windows functions, how would they compile?

sorry so many questions, i figure if you could just answer these, then i'd be ready to start googling stuff and what not

[Edited on December 22, 2005 at 4:04 PM. Reason : dsasd]

12/22/2005 4:04:02 PM

The functions are all called from code within your program.

Quote :

"seems unlikely, since the java compiler probably won't recognize the functions"

this is the problem.

JAVA is OS independent so it has no way to make calls to windows native functions.
If you used something like VB or one of the .net languages you can access the functions natively from within your code.

However, there is something called JNI (Java Native Interface) that allows your Java program to interact with the OS and any libraries compiled for the OS. More info Here

So you'd have to create your own java functions to interact with the windows API libraries via JNI. Someone out there has probably already done this for you though. So before you try to write it yourself i'd take a look.

But like i said. J# uses java syntax and you'd be able to use the Windows API functions natively without needing a JNI bridge.

12/22/2005 4:10:23 PM

much thanks

i actually think that is what the robot class is, it moves and clicks the cursor, and can enter text

now i just need to search around for one that can grab next from a read-only window, i'm sure it's out there, and if i can't find it i guess i can just model the robot implementation and make the necessary changes

[Edited on December 22, 2005 at 4:22 PM. Reason : dsf]

12/22/2005 4:21:52 PM

javascript on MY site displays the time in binary...take that!

12/22/2005 5:27:55 PM

use VB script or VB. it can do what you need. you can access API functions via a declare statement. VB is really easy to use. alternatively you can do it in C/C++. both are readily equipped for that sort of thing. you can use C#, or J# too. Java really isn't the language of choice in this case.

12/23/2005 3:54:55 PM

this would be easiest to do with VBScript/JScript/CScript

12/23/2005 8:07:57 PM

Quote :

"Java really isn't the language of choice in this case."

the reason i thought of java is because it is good with web applets and stuff. The host program is a gui that would be accessed via the internet, so i thought it seemed appropriate, i guess i might look into vb though

12/27/2005 4:50:34 PM

j# can be deployed using something similar to java web start.

12/27/2005 4:51:38 PM