Search Engine , Web Crawler

Hi,

I am trying to build a search engine.[Java]


I have made a web crawler which on being given an initial [starting ]
URL,

1.will go to that URL
2.Retrieve all the URL's present on that web page.
3.Store them in URL_Database_file.
4.After finishing with that URL it retrieves the next URL automatically from
URL_Database_file.[Then repeats steps 1-4].


Now i want some guidance regarding how to retrieve words from a web pages HTML code, ie:- <p>Sahil Ansari needs Help</p> and then store them in a file.:confused::confused:

You could read the HTML file in from some sort of InputStream and parse it from there.

This is a natch ( naturally acclimatized ) problem domain for REGEX - I need regex practice and will gladly help. Try, for starters, making a Pattern to look for > and <

You will get stuck right away on what is called escape sequences. Doing good regex patterns for html work is substantially harder than drinking coffee.

Hi Nicholas Jordan,

Well i have already done what u said,
> <, and even after doing this we still need to separate the words individually, it's not easy but i have done that too.
There will be problem if tags open & close on different lines. <p>.....
.....</p>

I am looking for a more sophisticated tool or method if possible.
This was my final year project B.Tech[Computer Science & Engineering] project.[submitted in May 2008]

Now i am just trying to improve upo it.
I have some idea, please tell if it's possible :- a web browser like internet explorer also reads an html code & displays the content, so it has already done all the hard work of ><,

now instead of displaying it if we can make it to write to file we will get what we want.

Hi,
parsing is not easy, coz HTML code on web does not have a fixed syntax.

New tags are added to html
Tags may not be closed
Multiline opening & closing tags.
tags emedded within other tags upto N levels.

Read my reply given to
Nicholas Jordan Quote

This code here was created using Jeffrey E.F. Friedl's Mastering Regular Expressions by O'Reilly. This is a first scratch and no attempt was made for a compile. Regexs are how it's done. Get over it.

Code:

---

import java.util.*;//
import java.net. ;//
import java.util.regex.Pattern;//
import java.util.regex.Matcher;//

public class LinkGrabber
{
    private int LinkCount = 0x0000;
    private List<URL> EarlsURLs = new LinkedList<URL>();
    private Pattern PickerPattern;
    public LinkGrabber(URL firstURL )
    {
        EarlsURLs.addFirst(firstURL);
        // Addional work later possibly.
        PickerPattern = Pattern.compile("<([^\"]*\"|'[^']*'|[^'\">])*>");//
    }
    public void parsePage()
    {
        ;
    }
}

---