Accessing Data From Websites in Java

[capitalistpig]

I am pretty new to Java, so I'm sorry if this is in the wrong place or is a stupid question. Can I get a simple Java console program to access a website and retrieve data to store as a variable? I want a program that can log onto finance.yahoo.com or some other site and retrieve information about the stock market. Is there a relatively simple way to do this?

An example would be ronpaulgraphs.com. He has a java program that would go to Ron Paul's website and get the donation total every 20 minutes and then graph it.

[gcalvin]

You certainly can do what you're talking about, but there are definite disadvantages to this approach. It's called "scraping" and it ties your app tightly to the current design and layout of the web site you are trying to scrape. If it's stock information you're interested in, you're probably better off using one of the many stock quote web services that are out there. (Note that scraping finance.yahoo.com is very likely a violation of their Terms of Service.)

If you really want to do web site scraping, you'll want to use the Parser class:

Parser (Java Platform SE 6)

If you decide to go with the web service approach, you'll want to read the Web Services tutorial:

The WSIT Tutorial

-Gary-

[Eranga]

If it's stock information you're interested in, you're probably better off using one of the many stock quote web services that are out there.

Yeah. It's much easier and much safer.

Note that scraping finance.yahoo.com is very likely a violation of their Terms of Service.

That's why such valuable resources provide services to access legal information, up to a certain level.

[capitalistpig]

I'm sorry, but that web service tutorial is a bit above my head. Are there any pre-made java projects in eclipse that I could download and play around with? All I know about java is how to write a "hello world" application and basic stuff with variables. Is there anywhere I could get the code for a very basic java program that retrieves stock quotes?

Also, are there any websites that would let me scrape them? What I really want is not the stock quotes, but the data like what is found here: http://finance.yahoo.com/q/ks?s=AAPL+Key+Statistics . Would I get in trouble for scraping yahoo? I doubt web services would have information that is that detailed. Thanks!

[gcalvin]

I sympathize, and I have been where you are, but there really aren't any shortcuts to where you want to go. I understand that web services seem complex and difficult to understand when you haven't used them before, but believe me, they are much simpler than scraping a web site in the long run.

If you're just playing around with the finance.yahoo.com data for your own use, I doubt they'd notice. If you're trying to build a business around it, they'll notice for sure, and it will be easy for them to shut you down. Note that they get their financial data from Capital IQ, which is owned by Standard & Poor's. They're all about making the data available to developers, but they're going to want to be paid for it.

I think you simply need to learn more Java skills before you try to tackle this project. Third-party data integration is never as simple as it looks, and is not for beginners. I'm not trying to discourage you -- quite the opposite, in fact -- but there aren't any shortcuts, so don't feel bad that it takes you so long to learn this stuff. It takes everybody a long time.

-Gary-

[Eranga]

I think you simply need to learn more Java skills before you try to tackle this project. Third-party data integration is never as simple as it looks, and is not for beginners. I'm not trying to discourage you -- quite the opposite, in fact -- but there aren't any shortcuts, so don't feel bad that it takes you so long to learn this stuff. It takes everybody a long time.

True. Build up a scrapper with Java is not an easy task for a begging. With skills of our OP it's may be much harder. If you are a quick learner you can looking into some example, but still need a better understanding about the basis.

[capitalistpig]

Would anyone be willing to build a simple program in eclipse that would scrape AAPL: Key Statistics for Apple Inc. - Yahoo! Finance ? I can't afford much, but I would be willing to spend $20 or so.

If not, I'll just have to learn it.

[PhHein]

Learn it. Nobody will do it for $20.

[capitalistpig]

Damn. That sucks. How about $30?

[PhHein]

My hourly rate is $150, go figure :) You might want to ask on rentacoder.

[Lil_Aziz1]

I think you simply need to learn more Java skills before you try to tackle this project. Third-party data integration is never as simple as it looks, and is not for beginners. I'm not trying to discourage you -- quite the opposite, in fact -- but there aren't any shortcuts, so don't feel bad that it takes you so long to learn this stuff. It takes everybody a long time.

-Gary-

I definitely agree. You should really try doing this project without any assistant. Remember, “Experience is what you get when you don't get what you want,” (Dan Stanford).

[curmudgeon]

My hourly rate is $150, go figure :) You might want to ask on rentacoder.

I can do it for $10 less than PhHein, but I'll probably take twice as long, and don't offer the same money-back guarantee that he gives. :rolleyes:

[capitalistpig]

Damn, $150/hour ain't bad. That makes me want to learn programming. Sure beats mowing lawns and walking dogs!

Oh well, I guess I'll try to learn it. Where should I start? All I know is basic stuff with variables. The Web Service tutorial someone posted is way too complicated for me. Where should I start and what do I need to read to work up to that Web Service tutorial?

[Lil_Aziz1]

I'd start by reading Sun's comprehensive tutorials on the basics. The Java™ Tutorials Take e-notes, do some practice problems relevant to the tutorial you're reading, etc. And yea it'll take a while but it's worth the climb. (sadd.. quoting Miley Cyrus)