Results 1 to 3 of 3
  1. #1
    JimmyD is offline Member
    Join Date
    Oct 2011
    Location
    New Jersey
    Posts
    44
    Rep Power
    0

    Default Using Java to Download Webpages

    I've had this question for a long time, regarding using java to download and process websites.

    The task itself is simple, I have to do a login, and download a page after login. The package i use is apache.httpclient

    I used to use Charles to look at what data I submit to forms.

    This approach works fine for simple websites. However, for a little bit more advanced website, they using https, and I started to lose track when I used Charles to watch data transmission, there are a dozens data/pages poping out. I tried a few naive data posting and can't seem to log in with success.

    I watched someone else using a excel VBA script to do the exact same task. VBA used an IE api, basically VBA commands IE to do the loggin. That's very easy.

    I understand that with java, it's sort of like java is trying to implement a working browser itself, that's why I failed because I don't know what data to send.

    Anyone has any insights on this topic?

    One approach I am considering is to find a java api to a browser (IE or Mozilla), but it seems nobody even talks about it. my guess is java network programming is just so basic that it does not need an api with a browser to do the job?

    Or any suggestion? any software better than charles?

  2. #2
    DarrylBurke's Avatar
    DarrylBurke is offline Member
    Join Date
    Sep 2008
    Location
    Madgaon, Goa, India
    Posts
    11,197
    Rep Power
    19

    Default Re: Using Java to Download Webpages

    Moved from Advanced Java

    db
    If you're forever cleaning cobwebs, it's time to get rid of the spiders.

  3. #3
    JimmyD is offline Member
    Join Date
    Oct 2011
    Location
    New Jersey
    Posts
    44
    Rep Power
    0

    Default Re: Using Java to Download Webpages

    any body can give me a hand on this?

Similar Threads

  1. Deployment of JSP and other language webpages on same server???
    By anup1710 in forum JavaServer Pages (JSP) and JSTL
    Replies: 2
    Last Post: 02-01-2011, 04:48 PM
  2. Download java 3d lib
    By Dennis in forum Java 2D
    Replies: 43
    Last Post: 09-15-2010, 10:05 AM
  3. Replies: 2
    Last Post: 09-29-2009, 03:33 PM
  4. how to download using java
    By leonard in forum Advanced Java
    Replies: 1
    Last Post: 08-06-2007, 05:36 PM
  5. Java and Webpages
    By maniaco in forum New To Java
    Replies: 1
    Last Post: 07-25-2007, 08:20 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •