Using Java to Download Webpages

[JimmyD]

I've had this question for a long time, regarding using java to download and process websites.

The task itself is simple, I have to do a login, and download a page after login. The package i use is apache.httpclient

I used to use Charles to look at what data I submit to forms.

This approach works fine for simple websites. However, for a little bit more advanced website, they using https, and I started to lose track when I used Charles to watch data transmission, there are a dozens data/pages poping out. I tried a few naive data posting and can't seem to log in with success.

I watched someone else using a excel VBA script to do the exact same task. VBA used an IE api, basically VBA commands IE to do the loggin. That's very easy.

I understand that with java, it's sort of like java is trying to implement a working browser itself, that's why I failed because I don't know what data to send.

Anyone has any insights on this topic?

One approach I am considering is to find a java api to a browser (IE or Mozilla), but it seems nobody even talks about it. my guess is java network programming is just so basic that it does not need an api with a browser to do the job?

Or any suggestion? any software better than charles?

[DarrylBurke]

Moved from Advanced Java

db

[JimmyD]

any body can give me a hand on this?