Results 1 to 6 of 6

Thread: crawling

  1. #1
    Mrs. Deswal's Avatar
    Mrs. Deswal is offline Member
    Join Date
    Apr 2009
    Location
    Gurgaon,India
    Posts
    24
    Rep Power
    0

    Default crawling

    we had made a crawler and i want to crawl the as soon as they are updated
    can any one know how can i know when the site was last updeted
    Sharing knowledge means gaining more knowledge.:)

  2. #2
    dswastik is offline Senior Member
    Join Date
    Dec 2008
    Location
    Kolkata
    Posts
    286
    Rep Power
    10

    Default

    Have a look on getLastModified() of URLConnection class.

  3. #3
    Mrs. Deswal's Avatar
    Mrs. Deswal is offline Member
    Join Date
    Apr 2009
    Location
    Gurgaon,India
    Posts
    24
    Rep Power
    0

    Default recrawl

    But i want to know after what interval web site get updated and its the time to crawl again
    as i want the most updated news items.
    Sharing knowledge means gaining more knowledge.:)

  4. #4
    dswastik is offline Senior Member
    Join Date
    Dec 2008
    Location
    Kolkata
    Posts
    286
    Rep Power
    10

    Default

    The method returns the last modified time of the given url. So when you hit for the first time you get a time for e.g. time1. When you hit for the next time once again you get a time for e.g. time2. Now if time1==time2 -> nothing has been modified else -> some changes have taken place.

  5. #5
    Mrs. Deswal's Avatar
    Mrs. Deswal is offline Member
    Join Date
    Apr 2009
    Location
    Gurgaon,India
    Posts
    24
    Rep Power
    0

    Default

    Thanks for reply
    but most of the sites does not provide with exact modification time of their's.
    Sharing knowledge means gaining more knowledge.:)

  6. #6
    neeti is offline Member
    Join Date
    Jul 2009
    Posts
    13
    Rep Power
    0

    Default

    Do any one have any idea how one can know about the updation cycle of a website.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •