Results 1 to 5 of 5
  1. #1
    babbupandey is offline Member
    Join Date
    Jan 2010
    Posts
    3
    Rep Power
    0

    Default Creating a crawler for a particular website

    Hello everyone,

    I am trying to develop a crawler for a website (nextag.com), the idea is to take the search query and pass it on to nextag and then get the results of the website in an array.
    I am not sure even on how to get started, I tried reading tutorials but they were not much of a help for this particular problem. Any help will be greatly appreciated.

    Thanks

  2. #2
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,374
    Blog Entries
    1
    Rep Power
    18

  3. #3
    babbupandey is offline Member
    Join Date
    Jan 2010
    Posts
    3
    Rep Power
    0

    Default

    In a manner, yes.
    The website has its own search feature, however, I want that the user should pass the query through the program I have developed. I should be able to get total number of results and page-wise results in an array.
    So suppose user passes the parameter "mobile phone, 2" then I should be able to list the results given by NexTag.com for "mobile phone" on page number #2 of the results page.

  4. #4
    babbupandey is offline Member
    Join Date
    Jan 2010
    Posts
    3
    Rep Power
    0

    Default

    Is anybody coming to my rescue?

  5. #5
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,374
    Blog Entries
    1
    Rep Power
    18

    Default

    How about the following resources. I've customize few of them used several times. May those things not exactly what you want, but it's possible to modified them.

    Open Source Search Engines in Java

Similar Threads

  1. crawler doesn't start threads.
    By Pierced1 in forum Threads and Synchronization
    Replies: 2
    Last Post: 09-28-2009, 08:02 PM
  2. set up tomcat on website
    By hunterbdb in forum Java Servlet
    Replies: 1
    Last Post: 04-22-2009, 06:52 AM
  3. Writing to a website
    By Shawn in forum New To Java
    Replies: 6
    Last Post: 03-20-2009, 12:32 PM
  4. How to download website (Get all link in website)
    By finalmem in forum Advanced Java
    Replies: 0
    Last Post: 11-12-2008, 08:43 AM
  5. Search Engine , Web Crawler
    By sahil.ansari in forum Advanced Java
    Replies: 5
    Last Post: 07-21-2008, 01:53 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •