Web Spider - Extract URLS

[heveen]

Hello friends
I am totally new in Java programming. I have a module, 'Network Programming with Java' and i have to develop a simple web spider.

Can anyone suggest me how to do it?
Especially how will i extract urls from a spcific url.

Below is part of my code:
try
{
URL u = new URL("mainURL");
URLConnection uc = u.openConnection();
String contentType = uc.getContentType();

int contentLength = uc.getContentLength();

if (contentType.startsWith("text/") || contentLength == -1 )
{ BufferedReader in = new BufferedReader(new InputStreamReader(u.openStream()));
FileWriter out = new FileWriter ("home.html");

String str;

while ((str = in.readLine()) != null)
{

out.write(str);


How will i extract the urls from here..
I have try to manipulate the strings but in vain.
i just want to extract at least 5 links from the content i get.

Please help me in doing this.

thanks a lots

[serjant]

You need to write your own parsers, if it is HTML/PHP/ASP(Content-type: text/html) then java htmlparser library will help you, search on web, if it is a MS files, then POI library, if it is a pdf file then PDFBox or iText or JPOD will help you to do that.

[neeti]

Hi,
you can use regular expressions to extract urls from the text.