Unable to write/post to a URL from Java

[achab]

Hi,

I need to write 10101663 to the text field at treasurer.maricopa.gov/Parcel/Summary.aspx , hit Enter or Submit, then read the same URL after a few seconds.
The problem is, I need to do it from Java code.

I tried the code below, but it's not working. The data I read from treasurer.maricopa.gov/Parcel/Summary.aspx after trying to write 10101663 is the same data that was there all along, as if I didn't write anything.

Below is one of the variants of several pieces of code that I tried, all unsuccessful. Could you tell me what's missing from my code ?

import java.io.*;
import java.util.*;
import java.net.*;

public class WebWriteRead {
public static void main(String args[]) {
try {
URL taxIDURL = new URL("http://treasurer.maricopa.gov/Parcel/Summary.aspx");
HttpURLConnection con = (HttpURLConnection) taxIDURL.openConnection();
con.setDoInput (true);
con.setDoOutput (true);
con.setUseCaches (false);
con.setRequestMethod("POST");
con.setFollowRedirects(true);
String msg = URLEncoder.encode("10101663");
con.setRequestProperty("Content-Length", "" + msg.length());
con.setRequestProperty("Content-Language", "en-US");
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
DataOutputStream os = new DataOutputStream(con.getOutputStream());
// OutputStreamWriter osw = new OutputStreamWriter(os);
os.writeBytes(msg);
os.flush();
os.close();
// any response?
InputStreamReader isr = new InputStreamReader(con.getInputStream());
BufferedReader br = new BufferedReader(isr);
String line = null;
while ( (line = br.readLine()) != null)
System.out.println("line: " + line);


}
catch (MalformedURLException e) {
System.out.println("Malformed URL: http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
}
catch (IOException e) {
System.out.println("scanTreasurersWebSite - Caugh Exception " + e.toString());
}
}
}

[masijade]

Open the url in "Post" mode, get the "OutputStream" and write a "param=value" pair, then close the OutputStream and open the "InputStream" to get the response. Examine the HTML source of the site to find out what "param=Value" pair(s) to pass.

[achab]

Open the url in "Post" mode, get the "OutputStream" and write a "param=value" pair, then close the OutputStream and open the "InputStream" to get the response. Examine the HTML source of the site to find out what "param=Value" pair(s) to pass.

Hi Masijade,

Thanks for the reply. How do I open the URL in post mode. Doesn't
con.setDoOutput (true);
accomplish that ? If not, which line of code should I change or add to open the URL in "Post" mode ?

As far as "param=Value", the relevant section of the HTML source is:

<input name="ctl00$cphMainContent$txtParcelNumber" type="text" maxlength="9" id="ctl00_cphMainContent_txtParcelNumber" style="width:90px;" />

<input type="submit" name="ctl00$cphMainContent$btnSubmit" value="Submit" onclick="javascript:WebForm_DoPostBackWithOptions( new WebForm_PostBackOptions(&quot;ctl00$cphMainContent $btnSubmit&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, false))" id="ctl00_cphMainContent_btnSubmit" />

Should I chose the name ctl00$cphMainContent$txtParcelNumber as param ? I tried both the name and the id yesterday, and neither worked. But something else may have been wrong with my code at that time too.

Abdenour

[masijade]

The con.setRequestMethod("POST"); (among other things) does that, but you are not writing a param=value pair you are writing only the value.

Look at what "javascript:WebForm_DoPostBackWithOptions" does. You may have "download" some included .js file to see it, but it is there and seemingly does some additional things.

[DarrylBurke]

Cross posted
OTN Discussion Forums : Unable to write/post to a URL from Java ...

Any more?

db

[masijade]

In answer to your reply on OTN that this was your only cross-post and your question as to whether DB means you should cross-post it more:

No, he, and all others here, think you shouldn't crosspost at all, at least not without providing links to those threads and a clear statement that it is crossposted. Otherwise, everyone trying to help you is wasting their time. As clearly exemplified here, as I wasted my time giving you the same advice EJP had already given you.

[achab]

The con.setRequestMethod("POST"); (among other things) does that, but you are not writing a param=value pair you are writing only the value.

Look at what "javascript:WebForm_DoPostBackWithOptions" does. You may have "download" some included .js file to see it, but it is there and seemingly does some additional things.

Hi Masijade,

I can't find an included .js file. I searched for the string include, and only found one instance (just regular text). I also only found one instance of .js, and it doesn't look at all like an include statement. It's inside a document.write() call.

Abdenour

[achab]

Speaking of cross posting, I did post another technical question on Java-Forums.org at
Could you tell me how to fix the offending 3 lines try block

Technically, it's a different question. No write/post needed. Just reading a web page. From a business perspective, I only need one of the 2 questions answered. I wonder though if the problem in that thread could give you insight into what the problem is in this thread. In the other thread, I am getting an exception when I try to get a DataInputStream for
treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663

If you manually type treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 in your browser (other thread's issue), or manually write 10101663 in treasurer.maricopa.gov/Parcel/Summary.aspx (this thread's issue), you get the exact same result. Maybe my inabilities to do either from Java are related in some way.


Abdenour

[masijade]

In the other thread, I am getting an exception when I try to get a DataInputStream for
treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663

If you manually type treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 in your browser (other thread's issue), or manually write 10101663 in treasurer.maricopa.gov/Parcel/Summary.aspx (this thread's issue), you get the exact same result. Maybe my inabilities to do either from Java are related in some way.

This may (probably) has more to do with the fact that the server attempts to determine if it is a browser doing the query or a program. Try setting the user agent (Google "Java user agent") to a typical browser user agent (google "internet explorer user agent").

Edit: In which case you can use the "get" version. And what you need to use in the post version is, of course, "Parcel=10101663".

[achab]

Is there a way to spoof the User agent in Java so the web server thinks it's Firefox calling it ? I have seen the question asked in some forums, but not answered.

Alternatively, in the "get" version, can my Java code have the Firefox on my machine read the web page, and then the Java code read the page source from my Firefox ? This alternative solution, if possible, might be unacceptably slow though. I eventually need to read over 10,000 of of such pages from my program and need that done in less than a 12 hours (and I have never done multi-thread programming before - so, for now at least, I plan to do that serially). So, if spoofing is possible, spoofing would be MUCH better. As would any solution where my Java successfully talks to the web server on the site from which I want to read.

[masijade]

Did you google what I told you to? You can find plenty of examples with that. Also, you can do the "get" version with an HttpURLConnection as well, no need to try involving an external app. I'm sorry, but, jebus, just try it.

[achab]

Did you google what I told you to? You can find plenty of examples with that. Also, you can do the "get" version with an HttpURLConnection as well, no need to try involving an external app. I'm sorry, but, jebus, just try it.

Hi Masijade,

I did. Here is my new try block in the "get/read" method:

try {
// System.setProperty("http.agent", "");
URL taxIDURL = new URL("http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
HttpURLConnection conn = (HttpURLConnection) taxIDURL.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)");
System.out.println ("conn.getResponsecode() = " + conn.getResponseCode());
DataInputStream data = new DataInputStream(new BufferedInputStream(conn.getInputStream()));
}

Still getting the exception. Also, conn.getResponseCode() is 400.


Jebus

[masijade]

400 is with authorisation. Do you have any sign in on that sight (maybe you use a Cookie). In any case google "Java HttpURLConnection authorisation". I'm sorry, but that is, currently, less a Java probelm (per se) than an understanding of HTTP. You first need to realise that it is about authorisation (and what that is and what it entails) before you can fix it and that requires an understanding of HTTP.

Of course "the exception" doesn't help much, knowing what exception from exactly which line, might. The ResponseCode seems to tell the story, maybe.

[achab]

400 is with authorisation. Do you have any sign in on that sight (maybe you use a Cookie). In any case google "Java HttpURLConnection authorisation". I'm sorry, but that is, currently, less a Java probelm (per se) than an understanding of HTTP. You first need to realise that it is about authorisation (and what that is and what it entails) before you can fix it and that requires an understanding of HTTP.

Of course "the exception" doesn't help much, knowing what exception from exactly which line, might. The ResponseCode seems to tell the story, maybe.

The site doesn't require any sign in. See for yourself. Type treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 in your browser and you will see that no sign in is required. And, if this is the first time you go to that site, no cookie was needed.

I use Firefox. It seems strange to me that Java code on my computer cannot get what Firefox can get from the same computer.

[masijade]

Sorry 401 is authorisation 400 is bad request.

And, BTW, Java has no problem with opening, reading, posting a web connection as long as it is done correctly.

A little while back I had something I was writing and wanted to internationalise it a bit and wrote a little utility program that read my properties and connected to a dictionary website providing arguments for the source language, the target language, and the word and got back the translated word (wrote multiple lines for those that had multiple translations so that I had to choose which) all without a hitch. So, everything you want to do here is more than possible, and is usually easily done.

You should probably be calling new HttpURLConnection and then all of those "set" statements and then call connect.

[achab]

WebScanner.java:11: java.net.HttpURLConnection is abstract; cannot be instantiated

[masijade]

Nevermind that. Just take this and see what you can get from it.

String item = URLEncoder.encode(tag, "UTF-8");

Properties systemSettings = System.getProperties();
systemSettings.put("proxySet", "true");
systemSettings.put("http.proxyHost", "some.proxy.host");
systemSettings.put("http.proxyPort", "80");

URL url = new URL("http://www.freedict.com/onldict/onldict.php");
sun.misc.BASE64Encoder encoder = new sun.misc.BASE64Encoder();  // there is now a better BASE64Encoder, BTW, I believe through the Preferences Classes
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
String encodedUserPwd = encoder.encode("domain\\user:xxxxxx".getBytes());
connection.setRequestProperty("Proxy-Authorization", "Basic " + encodedUserPwd);
connection.setRequestMethod("POST");
connection.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
try {
    out.write("search=" + item);
    out.write("&exact=true");
    out.write("&max=10");
    out.write("&from=English");
    out.write("&to=" + lang);
    out.write("&fname=" + fname);
    out.write("&back=" + back);
} finally { try { out.close(); } catch (Exception e) {} };
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
try {
    String decodedString;
    while ((decodedString = in.readLine()) != null) {
        if (decodedString.contains("result-r-blue")) {
            decodedString = decodedString.replaceFirst("^.*<(em|strong)>", "").replaceFirst("</(em|strong)>.*$", "");
            if (!decodedString.contains("No matches found")) {
                value = decodedString;
            }
        }
    }
} finally { try { in.close(); } catch (Exception e) {} }

[masijade]

WebScanner.java:11: java.net.HttpURLConnection is abstract; cannot be instantiated

I know, it had been a while though. I was already reposting when you posted.

[masijade]

P.S. that entire block of code should be wrapped in a try catch I simply didn't include that try, nor the catch block.

[achab]

Thanks Masijade for sharing your code. Part of it is related to writing/posting, which I ignored. I copied what might be related to reading. My new try block (listed below) still fails with the same error message. Here is one things that puzzles me. I ask the web server to open treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 , and, from the stacktrace, the web server thinks I am trying to open treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663

code:
try {
// System.setProperty("http.agent", "");
// String item = URLEncoder.encode(tag, "UTF-8");

Properties systemSettings = System.getProperties();
systemSettings.put("proxySet", "true");
systemSettings.put("http.proxyHost", "some.proxy.host");
systemSettings.put("http.proxyPort", "80");

CookieHandler.setDefault(new CookieManager());
URL taxIDURL = new URL("http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
HttpURLConnection conn = (HttpURLConnection) taxIDURL.openConnection();
conn.setRequestProperty("Proxy-Authorization", "Basic ");
conn.setRequestProperty("User-Agent", "MSIE");
conn.setFollowRedirects(true);
System.out.println ("conn.getResponsecode() = " + conn.getResponseCode());
BufferedReader data = new BufferedReader(new InputStreamReader(conn.getInputStream()));
}