Results 1 to 9 of 9
  1. #1
    mbschultz97 is offline Super OP Noob
    Join Date
    May 2014
    Location
    Virginia
    Posts
    65
    Rep Power
    0

    Default How to get info from sites...

    I was wondering where can I find info on how to grab info from websites to use in a program?
    Last edited by mbschultz97; 05-10-2014 at 06:39 PM.

  2. #2
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,879
    Rep Power
    25

    Default Re: How to get info from sites...

    Send an HTTP GET for the page to the site and use some scanning techniques to get your data from the lines returned by the server.
    If you don't understand my response, don't ignore it, ask a question.

  3. #3
    jashburn is offline Senior Member
    Join Date
    Feb 2014
    Posts
    219
    Rep Power
    1

    Default Re: How to get info from sites...

    Grabbing info (presumably in HTML form) from web sites is known as "web scraping". Here's an article by the author of a book on web scraping using Java: Creating a sample web scraper | Packt Publishing

    Note that this is commonly done with the help of a library, such as jsoup that is used in the article.

  4. #4
    JosAH's Avatar
    JosAH is offline Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,765
    Blog Entries
    7
    Rep Power
    21

    Default Re: How to get info from sites...

    Quote Originally Posted by mbschultz97 View Post
    I was wondering where can I find info on how to grab info from websites to use in a program?
    Don't expect too much of it: the only content you can get is that crappy html stuff, while most of those http servers make their clients (browsers) display the information (the stuff that you want) through Javascript and those darn XMLHttp thingies (AJAX). It's all messy stuff and not very well designed.

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  5. #5
    mbschultz97 is offline Super OP Noob
    Join Date
    May 2014
    Location
    Virginia
    Posts
    65
    Rep Power
    0

    Default Re: How to get info from sites...

    Quote Originally Posted by JosAH View Post
    Don't expect too much of it: the only content you can get is that crappy html stuff, while most of those http servers make their clients (browsers) display the information (the stuff that you want) through Javascript and those darn XMLHttp thingies (AJAX). It's all messy stuff and not very well designed.

    kind regards,

    Jos
    so it would be pretty hard to make a program that gets the temperature from some website?

  6. #6
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,879
    Rep Power
    25

    Default Re: How to get info from sites...

    Go to the site with a browser and View the source. That will give you an idea of what the JAVA code would have to parse.
    There are some sites that give small, usable responses for data like temperature. Not sure what that interface is called.
    If you don't understand my response, don't ignore it, ask a question.

  7. #7
    jashburn is offline Senior Member
    Join Date
    Feb 2014
    Posts
    219
    Rep Power
    1

    Default Re: How to get info from sites...

    For web scraping in general, if you can visually work out the data to be extracted by doing a web page view-source, then it's usually not difficult to programmatically extract the data. I've done this before using jsoup.

    For weather/temperature data specifically, you have a few more options that don't rely on web scraping. One option is to use a web-based API such as REST. See 5 Weather Forecast API for developing apps | Super Dev Resources. E.g., you can send a GET request to http://api.openweathermap.org/data/2...q=austin,texas to receive a response in JSON format for the weather (including temperature) in Austin, Texas.

    Another option is to use an RSS feed. E.g., if you go to BBC Weather and "Find a Forecast" for "Texas", on the result page you'll find an RSS icon next to the printer icon. The RSS icon contains the links to the RSS feeds for Texas City weather, e.g., BBC Weather - Observations for Texas City, United States of America for the current weather, and BBC Weather - Forecast for Texas City, United States of America for the 3-day forecast. These feeds are in XML format that can be programmatically parsed.

  8. #8
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,879
    Rep Power
    25

    Default Re: How to get info from sites...

    Continued at: URL help
    If you don't understand my response, don't ignore it, ask a question.

  9. #9
    JosAH's Avatar
    JosAH is offline Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,765
    Blog Entries
    7
    Rep Power
    21

    Default Re: How to get info from sites...

    Quote Originally Posted by mbschultz97 View Post
    so it would be pretty hard to make a program that gets the temperature from some website?
    It al depends on the website; no general advice or guidelines can be given ...

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

Similar Threads

  1. what are the best sites
    By chathura992 in forum New To Java
    Replies: 4
    Last Post: 12-30-2010, 04:58 PM
  2. High BW sites
    By Sarinam in forum Networking
    Replies: 2
    Last Post: 06-16-2008, 03:38 PM
  3. sharing of data between sites
    By jyovasinedu in forum Advanced Java
    Replies: 1
    Last Post: 11-02-2007, 09:56 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •