html web page parsing/scraping

[orchid]

Hi, I am trying to automate some routine web browsing functionality. I need to log in/enter information/etc...however, the part that gets tricky (as far as finding a solution) is at some point after submitting information from a page, the links returned are undetermined...in other words, the results are not always the same (as far as number of naming)...and I need a way of accessing the links returned...determining their text, and being able to continue to specific links from there...certain screen scrapers out there come very close to doing what i want with the exception of the last part. any java api out there to handle this type of stuff?? I've tried httpunit & something very similar (forget the name), but they didn't work...i think issues with java script, etc...looking for perhaps a language or java api specifically geared around this type of stuff...if anyone has any insight, i would greatly appreciate it!! thanks...

[derrickD]

For any given page you should use an HTML parser to parse and process the document in any way you see fit. This allows you to retreive all links etc. Apache also has some really nice libraries in the HTTPComponents sub project.
HTML Parser - HTML Parser
HttpComponents - HttpComponents Overview

Also, if you choose not to elect Java for the task, I would suggest Python.

[francojava1]

Hello dear orchid, Iam francojava1, who suggests you visit at this sample HTML Scraper Python recipes HTML Scraper « Python recipes « ActiveState Code. Please tell me what is the part of codes that want to codified there .Also, I could build a java API with respect to this parser Ok.

Thanks.

[Fubarable]

Perhaps we shouldn't wake up long dead posts. Locking.