Results 1 to 3 of 3
  1. #1
    amitvit014 is offline Member
    Join Date
    Nov 2011
    Posts
    4
    Rep Power
    0

    Default How to use Sitemap for webcrawling in JAVA

    I need to develop the crawler which will crawl all the product link and price from a site . I heard that its more efficient and fast to use Sitemap which is mention in robots.txt file but I am not able to figured out how to use sitemap for crawling. please someone help me how to use Sitemap for crawling.

  2. #2
    vinci123 is offline Member
    Join Date
    Aug 2011
    Location
    INDIA
    Posts
    64
    Rep Power
    0

    Default Re: How to use Sitemap for webcrawling in JAVA

    first do tell me what you actually want to develop?crawler or site map?In my sight i never found anyone saying to develop crawler(google crawler etc) but you can prepare sitemap in xml and html.How and where you have heard of sitemap in java?
    I can and I will

  3. #3
    DarrylBurke's Avatar
    DarrylBurke is offline Forum Police
    Join Date
    Sep 2008
    Location
    Madgaon, Goa, India
    Posts
    11,423
    Rep Power
    20

    Default Re: How to use Sitemap for webcrawling in JAVA

    The question was clear. The OP wants to develop a crawler.

    db
    If you're forever cleaning cobwebs, it's time to get rid of the spiders.

Similar Threads

  1. How to use Sitemap for WebCrawling
    By amitvit014 in forum Advanced Java
    Replies: 2
    Last Post: 11-15-2011, 09:20 PM
  2. [B]How to use Sitemap for WebCrawling[/B]
    By amitvit014 in forum New To Java
    Replies: 1
    Last Post: 11-15-2011, 10:46 AM
  3. Webcrawling www.stock-forecasting.com
    By devasia1000 in forum Networking
    Replies: 0
    Last Post: 02-12-2011, 11:25 AM
  4. webcrawling and harvesting
    By locke19 in forum New To Java
    Replies: 2
    Last Post: 04-14-2010, 02:53 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •