Results 1 to 6 of 6

Thread: crawling

  1. #1
    Mrs. Deswal's Avatar
    Mrs. Deswal is offline Member
    Join Date
    Apr 2009
    Location
    Gurgaon,India
    Posts
    24
    Rep Power
    0

    Default crawling

    we had made a crawler and i want to crawl the as soon as they are updated
    can any one know how can i know when the site was last updeted
    Sharing knowledge means gaining more knowledge.:)

  2. #2
    dswastik is offline Senior Member
    Join Date
    Dec 2008
    Location
    Kolkata
    Posts
    283
    Rep Power
    7

    Default

    Have a look on getLastModified() of URLConnection class.

  3. #3
    Mrs. Deswal's Avatar
    Mrs. Deswal is offline Member
    Join Date
    Apr 2009
    Location
    Gurgaon,India
    Posts
    24
    Rep Power
    0

    Default recrawl

    But i want to know after what interval web site get updated and its the time to crawl again
    as i want the most updated news items.
    Sharing knowledge means gaining more knowledge.:)

  4. #4
    dswastik is offline Senior Member
    Join Date
    Dec 2008
    Location
    Kolkata
    Posts
    283
    Rep Power
    7

    Default

    The method returns the last modified time of the given url. So when you hit for the first time you get a time for e.g. time1. When you hit for the next time once again you get a time for e.g. time2. Now if time1==time2 -> nothing has been modified else -> some changes have taken place.

  5. #5
    Mrs. Deswal's Avatar
    Mrs. Deswal is offline Member
    Join Date
    Apr 2009
    Location
    Gurgaon,India
    Posts
    24
    Rep Power
    0

    Default

    Thanks for reply
    but most of the sites does not provide with exact modification time of their's.
    Sharing knowledge means gaining more knowledge.:)

  6. #6
    neeti is offline Member
    Join Date
    Jul 2009
    Posts
    13
    Rep Power
    0

    Default

    Do any one have any idea how one can know about the updation cycle of a website.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •