Results 1 to 14 of 14
  1. #1
    capitalistpig is offline Member
    Join Date
    Jun 2010
    Posts
    5
    Rep Power
    0

    Default Accessing Data From Websites in Java

    I am pretty new to Java, so I'm sorry if this is in the wrong place or is a stupid question. Can I get a simple Java console program to access a website and retrieve data to store as a variable? I want a program that can log onto finance.yahoo.com or some other site and retrieve information about the stock market. Is there a relatively simple way to do this?

    An example would be ronpaulgraphs.com. He has a java program that would go to Ron Paul's website and get the donation total every 20 minutes and then graph it.

  2. #2
    gcalvin is offline Senior Member
    Join Date
    Mar 2010
    Posts
    952
    Rep Power
    5

    Default

    You certainly can do what you're talking about, but there are definite disadvantages to this approach. It's called "scraping" and it ties your app tightly to the current design and layout of the web site you are trying to scrape. If it's stock information you're interested in, you're probably better off using one of the many stock quote web services that are out there. (Note that scraping finance.yahoo.com is very likely a violation of their Terms of Service.)

    If you really want to do web site scraping, you'll want to use the Parser class:

    Parser (Java Platform SE 6)

    If you decide to go with the web service approach, you'll want to read the Web Services tutorial:

    The WSIT Tutorial

    -Gary-

  3. #3
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,371
    Blog Entries
    1
    Rep Power
    20

    Default

    Quote Originally Posted by gcalvin View Post
    If it's stock information you're interested in, you're probably better off using one of the many stock quote web services that are out there.
    Yeah. It's much easier and much safer.

    Quote Originally Posted by gcalvin View Post
    Note that scraping finance.yahoo.com is very likely a violation of their Terms of Service.
    That's why such valuable resources provide services to access legal information, up to a certain level.

  4. #4
    capitalistpig is offline Member
    Join Date
    Jun 2010
    Posts
    5
    Rep Power
    0

    Default

    I'm sorry, but that web service tutorial is a bit above my head. Are there any pre-made java projects in eclipse that I could download and play around with? All I know about java is how to write a "hello world" application and basic stuff with variables. Is there anywhere I could get the code for a very basic java program that retrieves stock quotes?

    Also, are there any websites that would let me scrape them? What I really want is not the stock quotes, but the data like what is found here: http://finance.yahoo.com/q/ks?s=AAPL+Key+Statistics . Would I get in trouble for scraping yahoo? I doubt web services would have information that is that detailed. Thanks!
    Last edited by capitalistpig; 06-15-2010 at 05:37 PM.

  5. #5
    gcalvin is offline Senior Member
    Join Date
    Mar 2010
    Posts
    952
    Rep Power
    5

    Default

    I sympathize, and I have been where you are, but there really aren't any shortcuts to where you want to go. I understand that web services seem complex and difficult to understand when you haven't used them before, but believe me, they are much simpler than scraping a web site in the long run.

    If you're just playing around with the finance.yahoo.com data for your own use, I doubt they'd notice. If you're trying to build a business around it, they'll notice for sure, and it will be easy for them to shut you down. Note that they get their financial data from Capital IQ, which is owned by Standard & Poor's. They're all about making the data available to developers, but they're going to want to be paid for it.

    I think you simply need to learn more Java skills before you try to tackle this project. Third-party data integration is never as simple as it looks, and is not for beginners. I'm not trying to discourage you -- quite the opposite, in fact -- but there aren't any shortcuts, so don't feel bad that it takes you so long to learn this stuff. It takes everybody a long time.

    -Gary-

  6. #6
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,371
    Blog Entries
    1
    Rep Power
    20

    Default

    Quote Originally Posted by gcalvin View Post
    I think you simply need to learn more Java skills before you try to tackle this project. Third-party data integration is never as simple as it looks, and is not for beginners. I'm not trying to discourage you -- quite the opposite, in fact -- but there aren't any shortcuts, so don't feel bad that it takes you so long to learn this stuff. It takes everybody a long time.
    True. Build up a scrapper with Java is not an easy task for a begging. With skills of our OP it's may be much harder. If you are a quick learner you can looking into some example, but still need a better understanding about the basis.

  7. #7
    capitalistpig is offline Member
    Join Date
    Jun 2010
    Posts
    5
    Rep Power
    0

    Default

    Would anyone be willing to build a simple program in eclipse that would scrape AAPL: Key Statistics for Apple Inc. - Yahoo! Finance ? I can't afford much, but I would be willing to spend $20 or so.

    If not, I'll just have to learn it.

  8. #8
    PhHein's Avatar
    PhHein is offline Senior Member
    Join Date
    Apr 2009
    Location
    Germany
    Posts
    1,429
    Rep Power
    7

    Default

    Learn it. Nobody will do it for $20.
    Math problems? Call 1-800-[(10x)(13i)^2]-[sin(xy)/2.362x]
    The Ubiquitous Newbie Tips

  9. #9
    capitalistpig is offline Member
    Join Date
    Jun 2010
    Posts
    5
    Rep Power
    0

    Default

    Damn. That sucks. How about $30?

  10. #10
    PhHein's Avatar
    PhHein is offline Senior Member
    Join Date
    Apr 2009
    Location
    Germany
    Posts
    1,429
    Rep Power
    7

    Default

    My hourly rate is $150, go figure :) You might want to ask on rentacoder.
    Math problems? Call 1-800-[(10x)(13i)^2]-[sin(xy)/2.362x]
    The Ubiquitous Newbie Tips

  11. #11
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    Quote Originally Posted by gcalvin View Post
    I think you simply need to learn more Java skills before you try to tackle this project. Third-party data integration is never as simple as it looks, and is not for beginners. I'm not trying to discourage you -- quite the opposite, in fact -- but there aren't any shortcuts, so don't feel bad that it takes you so long to learn this stuff. It takes everybody a long time.

    -Gary-
    I definitely agree. You should really try doing this project without any assistant. Remember, “Experience is what you get when you don't get what you want,” (Dan Stanford).
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

  12. #12
    curmudgeon is offline Senior Member
    Join Date
    May 2010
    Posts
    436
    Rep Power
    5

    Default

    Quote Originally Posted by PhHein View Post
    My hourly rate is $150, go figure :) You might want to ask on rentacoder.
    I can do it for $10 less than PhHein, but I'll probably take twice as long, and don't offer the same money-back guarantee that he gives. :rolleyes:

  13. #13
    capitalistpig is offline Member
    Join Date
    Jun 2010
    Posts
    5
    Rep Power
    0

    Default

    Damn, $150/hour ain't bad. That makes me want to learn programming. Sure beats mowing lawns and walking dogs!

    Oh well, I guess I'll try to learn it. Where should I start? All I know is basic stuff with variables. The Web Service tutorial someone posted is way too complicated for me. Where should I start and what do I need to read to work up to that Web Service tutorial?

  14. #14
    Lil_Aziz1's Avatar
    Lil_Aziz1 is offline Senior Member
    Join Date
    Dec 2009
    Location
    United States
    Posts
    343
    Rep Power
    5

    Default

    I'd start by reading Sun's comprehensive tutorials on the basics. The Java™ Tutorials Take e-notes, do some practice problems relevant to the tutorial you're reading, etc. And yea it'll take a while but it's worth the climb. (sadd.. quoting Miley Cyrus)
    "Experience is what you get when you don't get what you want" (Dan Stanford)
    "Rise and rise again until lambs become lions" (Robin Hood)

Similar Threads

  1. JNI accessing non primitive data type
    By H_P in forum Advanced Java
    Replies: 1
    Last Post: 04-14-2010, 06:43 AM
  2. Best Websites to learn Java?
    By Meta in forum New To Java
    Replies: 21
    Last Post: 03-18-2010, 10:45 AM
  3. Accessing Data from a .txt file
    By Oasis13 in forum New To Java
    Replies: 5
    Last Post: 02-01-2008, 01:16 AM
  4. Packaging and accessing data files
    By todd in forum Advanced Java
    Replies: 1
    Last Post: 08-01-2007, 01:27 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •