Developing a front end for a public website

[duphus]

I was asked to write a front end application that would pull data from the following website.

Atlas of Genetics and Cytogenetics in Oncology and Haematology

Specification is that it the application should be able to send a search term and return details about the query from the website.

I have done some research on HTML parsers and feel that this would be the first step but I am new to java and require some input.

Can someone please point me as to where to begin?

[quad64bit]

I've done quite a bit of this -- some sites are easier than others. In the best case, you can create a BufferedReader and give it a StreamReader that opens a data stream to a url you specify. The data that you get from the reader is the result of the URL being sent to the server -- so, if you send something simple like 'http://www.apple.com' you get the HTML for the Apple homepage.

However, you can also put javascript or php or html post/get messages in the url. In this case, the results you get are the results of the the 'query' you constructed in the initial URL.

Now, things get tricky when the webserver on the other end does not like that fact that you are not running a real web browser, rather a java app. In these cases (IMDB.com for example), you have to create a custom HTTP request that lies to the server and tells it your app is really Mozilla or something similar.

I've done it both ways, happy to share -- Why not tell us more about what progress you've made thus far?

[duphus]

Ok, so all I've done so far on research on html parsers. I'm undecided whether just to parse the whole website and dump into a database or try to figure out the post/get coding. It seems just to send a simple query isn't too bad to figure out but can you post some examples?

The problem is that the website only lets you search by gene name, whereas I might need to search each individual gene html file content for a gene position and construct complex queries later on. So it makes sense just to do a database dump to SQL. However, the site will be continously updated and I can't tax their server, they are french and may get upset. How is this usually done - just automate the dump every so often?

Any suggestions?