Results 1 to 3 of 3
  1. #1
    atomant is offline Member
    Join Date
    May 2010
    Posts
    45
    Rep Power
    0

    Default Scraping Data From a Website

    I'm doing research for school and I need to catalog article headlines from wsj.com. I'm using the search function at wsj.com to search for "google" in 2010. I know how to scrape data from a page using Java, but my problem is the url used in the wsj.com search. The url does not contain the advanced search options.

    So, a search url for "google" on wsj.com looks like this: "http://online.wsj.com/search/term.html?KEYWORDS=google", but if I try to use the advance features to narrow the search to 2010 and only articles, it is the same url.

    I use the httpclient from apache to get the site at a url. Is there some way to send the request with the advanced search features?

  2. #2
    atomant is offline Member
    Join Date
    May 2010
    Posts
    45
    Rep Power
    0

    Default Re: Scraping Data From a Website

    So, after looking through the source of the page I've come up with this: "http://online.wsj.com/search/term.html?KEYWORDS=goog&media=Articles&date_range= 4years&fromDate=01/01/10&toDate=12/31/10"

  3. #3
    quad64bit's Avatar
    quad64bit is offline Moderator
    Join Date
    Jul 2009
    Location
    VA
    Posts
    1,323
    Rep Power
    7

    Default Re: Scraping Data From a Website

    Yeah that sounds about right. Working for you now?

Similar Threads

  1. Web Scraping, web mining
    By mitukhan in forum Java Software
    Replies: 0
    Last Post: 09-19-2011, 01:44 PM
  2. Website to pull data from java sockets?
    By josephjah in forum Advanced Java
    Replies: 2
    Last Post: 07-24-2011, 08:24 AM
  3. how to post data to a website using java
    By mr_anderson in forum Advanced Java
    Replies: 6
    Last Post: 11-12-2010, 12:46 PM
  4. How to download data from a website.
    By bobrenjc93 in forum Java Applets
    Replies: 2
    Last Post: 05-15-2009, 08:08 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •