Results 1 to 3 of 3

Thread: web crawler

  1. #1
    jharishabh7 is offline Member
    Join Date
    Oct 2010
    Posts
    12
    Rep Power
    0

    Default web crawler

    Hi,
    I am developing a web crawler using java. I have implemented it to some extent, like I have developed program which parses all the hyperlinks from the entered URL and and visits each link one by one and iterates this process. Now I want to parse all the visible text from a particular web page. I am facing problem in this. Can anyone suggest how to accomplish this. Any help wil be greatly appreciated.
    Thanks in advance
    Rishabh jha

  2. #2
    Desh Banks is offline Member
    Join Date
    Nov 2010
    Posts
    12
    Rep Power
    0

    Default

    I suppose it depends on what you want to do to the text, but you can look for certain tags that specify webpage text.

    <p>, <table><othertags>...</table></othertags> etc... This might be a good application of regular expressions.

    What is the overall goal? To store the parsed text into tables/data structures in some ordered way?

    DBanks

  3. #3
    jharishabh7 is offline Member
    Join Date
    Oct 2010
    Posts
    12
    Rep Power
    0

    Default

    Dear Desh Banks,
    Thanks a lot for your concern, but I have already solved this problem, if you want to help, then please help me in parsing of all the visible text from a particular webpage after retrieving the source code.
    Regards,
    Rishabh Jha

Similar Threads

  1. Web crawler or RSS Feed
    By lokesh0011 in forum Java Servlet
    Replies: 1
    Last Post: 05-08-2010, 11:52 AM
  2. ChilKat web crawler
    By ishdevil in forum New To Java
    Replies: 0
    Last Post: 02-15-2010, 11:02 AM
  3. Creating a crawler for a particular website
    By babbupandey in forum Networking
    Replies: 4
    Last Post: 01-02-2010, 04:10 AM
  4. crawler doesn't start threads.
    By Pierced1 in forum Threads and Synchronization
    Replies: 2
    Last Post: 09-28-2009, 09:02 PM
  5. Search Engine , Web Crawler
    By sahil.ansari in forum Advanced Java
    Replies: 5
    Last Post: 07-21-2008, 02:53 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •