Parsing a web page: imitating a browser as cllse as possible

[LucasH]

Hi guys!
I'm creating a piece of software that basically crawls through several pages getting ALL the links on the page.
You may say "yeah, that's easy... you can use HtmlParser, JerichoParser, etc.", but the problem is that those are decent parsers for HTML links but VERY SLOW parsers for links on JavaScript scripts and other technologies.
That's the reason of my title. I need to crawl through ALL of the links on a page, but on a decent speed... JUST as a browser does.

My questions are:
1. Recommendations on a parser to do this job?
2. Which variety of links and technologies will I encounter (besides HTML and JS)? I need a parser that handles all of them on an efficient way.


Thanks in advance!
-Lucas

[LucasH]

can anyone lend me a hand on this one? :S

[LucasH]

anyone knows WHERE can I look for help regarding this issue?
thanks in advance

[Tolls]

How will you identify links in Javascript?

[LucasH]

Hi Tolls
Right now, I'm using JerichoParser to do that job... but it's too slow.
With Jericho I basically open all <script type="text/javascript"> src attribute.

[Tolls]

What about ones that are built up?
Or held as constants in chunks?

Just trying to figures out how far you are expecting to go with this.

[LucasH]

Can you give an example? Im not sure I get your point

[Tolls]

var SOME_TARGET = "something.action";

later on (in some other js file):
callAjax(SOME_TARGET + "<a load of parameters>");

This is possibly why Jericho takes so long.
Your browser only has to translate that at the point it's called, but a web page might have a ton of code that does this sort of thing that Jericho then has to figure out.

[JosAH]

I think it's an undecidable problem because you actually have to execute/interpret the Javascript code in order to find out what the actual links are. It is equivalent to a Turing Halting Problem and you don't want to go there.

kind regards,

Jos

[DarrylBurke]

Spammy link removed. ValeryB, there's a section of the forums for advertising. This isn't it.

db