Unable to write/post to a URL from Java

[masijade]

It seems to be using some homegrown sessions and looking at some of the HTML source there are a few hidden fields that are passed back and forth and those probably control the sessions. Look at the source again (and that one javascript method is contained in that page) and see what other fields you might be able to pass along. The "mechanics" of this seem to be correct, it seems, now, as though you simply need to find the right information to post. If need be get a packet sniffer and make the connection with the browser and then examine the network traffic that resulted and see what is really going on.

[achab]

For now, I have given up trying to post anything to the site ... only trying to read. Do you still think the source and included Javascript will help when all I am trying to do is read the HTML page that the web site is creating ?

[masijade]

Whether with "Get" or "Post" you are still providing information. And that site seems to run through a couple of redirects (building up a session as it goes, seemingly). I still think your best bet would be to go to the site where you have to enter the number, then start a packet sniffer and then enter a number submit the form and find out exactly what it is that the form submits and recreate (whether with get or post probably won't matter).

[achab]

Thanks for the tip. I will try that tomorrow. My brain batteries run out for tonight.

[achab]

Here is the first HTTP line from WireShark. There are other lines, but let's start with the first:
Protocol = HTTP
Info = GET /Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S HTTP/1.1

Based on the above, I added the following to my code:
String msg = "pn=" + URLEncoder.encode("10101663");
msg += "&tcd=" + URLEncoder.encode("1");
msg += "&fpn=" + URLEncoder.encode("101-01-663%201");
msg += "&mry=" + URLEncoder.encode("2010");
msg += "&sec=" + URLEncoder.encode("True");
msg += "&rf=" + URLEncoder.encode("False");
msg += "&dp=" + URLEncoder.encode("S");

msg is what I ultimately send to the web server.

How would you translate the "/Parcel/SetSession.asp?" part into Java code ?

It takes a few seconds for the web server to come back with the information I need. Should I make my program sleep a few seconds, or is there a smarter way to detect when the web server is ready to serve me the information I need ?

Abdenour

[masijade]

http://<server>/Parcel/SetSession.asp

With the rest of that as the part written to the outputstream in the post call.

As far as sleeping, no, simply attempt to open the inputstream and it will "wait" until it gets some data, or until the server closes it.

But this "rf=False$dp=S" is

msg += "&rf=" + URLEncoder.encode("False$dp=S");

and not

msg += "&rf=" + URLEncoder.encode("False");
msg += "&dp=" + URLEncoder.encode("S");

if that is really a "$" and not an "&".

BTW, that String is already URL encoded (%20 is " "), so there is no reason to call encode at all, and if you do change

msg += "&fpn=" + URLEncoder.encode("101-01-663%201");

to

msg += "&fpn=" + URLEncoder.encode("101-01-663 1");

[masijade]

Or, if you want to use a get simply use

"http://treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S"

as the url

[achab]

Oops ... it's rf=False&dp=S
I changed 101-01-663%201 to 101-01-663 1 in the write/post version.

With the new URL, the "get/read" version of the code no longer throws an exception. But, both the "get/read" and the "write/post followed by read" versions of the code give me an HTTP page source that contains "Maricopa County Parcel Inquiry", which is the page source you get if you go to treasurer.maricopa.gov/Parcel/Default.aspx and don't type anything.

As for the input stream waiting until it gets some data, when you do it manually, there is data there before you type anything. It just happens that the data I want is the one after you type, hit Submit, and wait a few seconds. Making the code sleep 5 seconds before getting in the InputStream doesn't solve the problem though.

Abdenour

[achab]

Here is the new full version of the "Get/Read" version of the code, which still doesn't work.

import java.io.*;
import java.util.*;
import java.net.*;

public class WebScanner {
    public static void main(String args[]) {
        try {
//            URL taxIDURL = new URL("http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
            URL taxIDURL = new URL("http://treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S");
            HttpURLConnection conn = (HttpURLConnection) taxIDURL.openConnection();
            conn.setRequestProperty("Proxy-Authorization", "Basic ");
            conn.setRequestProperty("User-Agent", "MSIE");
            conn.setFollowRedirects(true);
            BufferedReader data = new BufferedReader(new InputStreamReader(conn.getInputStream()));
            for ( int i=0 ; i<45 ; i++ ) {
                String line = data.readLine();
                System.out.println ("line = " + line);
            }
        }
        catch (MalformedURLException e) {
            System.out.println("Malformed URL: http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
        }
        catch (IOException e) {
            System.out.println("scanTreasurersWebSite - Caugh Exception " + e.toString());
            e.printStackTrace();
        }
    }
}

[masijade]

push "QUOTE" on one of my posts with code tags and you will see them.

[achab]

Thanks. I just did it. So the progress in the "Get/Read" version from yesterday is that there is no longer an exception thrown. The progress that needs to be made is to get the same HTTP one would get manually from typing:
treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S
into a web browser and waiting 1 second or so.

[masijade]

When I try it I get a "server redirected too many times" error. The server seems to be actively attempting to block automated lookups. I have the feeling the problem is at the server side.

BTW

conn.setFollowRedirects(true);

should be

HttpURLConnection.setFollowRedirects(true);

and should be done before calling openConnection.

[achab]

Thanks Masijade. I changed that. The problem still remains though. The output is the same: the page source of what you would get if you went to treasurer.maricopa.gov/Parcel/Default.aspx

I just rerun WireShark, this time simulating the "Get/Read" version, that is, hitting enter on my browser's URL, in which I had typed treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S before starting sniffing.

Here is the Info for first HTTP and HTTP/XML lines in WireShark. I highlighted in Bold the few lines that I think are relevant:

GET /root.sxml HTTP/1.1
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
GET /WANIPConn1.xml HTTP/1.1
HTTP/1.1 200 OK
POST /sqm/windowsLive/sqmserver.dll HTTP/1.1
HTTP /1.1 403 .6 Forbidden (text/html)
GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship
HTTP/1.1 301 Moved Permanently
GET /capacity HTTP/1.1
HTTP/1.1 200 OK (text/plain)
GET /capacity HTTP/1.1
HTTP/1.1 200 OK (text/plain)
GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship/~ts-101104/~l-en/config.xml HTTP/1.1
HTTP/1.1 200 OK
GET /Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S HTTP/1.1
HTTP/1.1 302 Object moved (text/html)
GET /Parcel/Summary.aspx HTTP/1.1
GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship
HTTP/1.1 200 OK (text/plain)
GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship/~ts-101104/~l-en/config.xml HTTP/1.1
HTTP/1.1 200 OK
GET /_utm.gif?utmwv=4.8.6&utmn=1516772251&utmhn=treasur er.maricopa.gov&utmcs=UTF-8&utmsr=1280x800&utmcs=32-bit&utmul=en-us&utmje=1&utmfl=..


I can't read the full last line above. How would you translate it into code ?

[achab]

The con.setRequestMethod("POST"); (among other things) does that, but you are not writing a param=value pair you are writing only the value.

Look at what "javascript:WebForm_DoPostBackWithOptions" does. You may have "download" some included .js file to see it, but it is there and seemingly does some additional things.

It seems like javascript:WebForm_DoPostBackWithOptions is a standard JavaScript method/function, not something defined in an included .js file. Is there an easy way to integrate Java with JavaScript, so that my code calls javascript:WebForm_DoPostBackWithOptions ?

Abdenour

[masijade]

No, it's not a standard. If you look at the source of the page where you enter the number it is contained at the top of the page between SCRIPT tags.

[achab]

Or, if you want to use a get simply use

"http://treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S"

as the url

No, it's not a standard. If you look at the source of the page where you enter the number it is contained at the top of the page between SCRIPT tags.

I have done a search through Ctrl-F, and don't see it. Here are the script tags:

<script type="text/javascript">
//<![CDATA[
var theForm = document.forms['aspnetForm'];
if (!theForm) {
    theForm = document.aspnetForm;
}
function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}
//]]>
</script>

.......

<script src="/Parcel/WebResource.axd?d=82X_DmeRwShoLJobNHsQnbVOlScs8_yWxC9DCuLI2YS__W5fmnbySsuN6RVESgJ05_dGX5bZ3fQBxx2amMPSDiePiBQ1&amp;t=634234388291823091" type="text/javascript"></script>

.................

<script src="/Parcel/WebResource.axd?d=0lTb8rR5d7hmnuOluNU1bbXvkHvoFQ_jnwji6bwN6BWAUT5z66CJqXrS0vVoz3RIP0OL323hu9vmTevPzdVyoW8A-7U1&amp;t=634234388291823091" type="text/javascript"></script>

................

<script type="text/javascript">
//<![CDATA[
WebForm_AutoFocus('ctl00_cphMainContent_txtParcelNumber');//]]>
</script>
</form>
 <script type="text/javascript">
	var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
	document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
	var pageTracker = _gat._getTracker("UA-2900680-1");
	pageTracker._initData();
	pageTracker._trackPageview();
</script>

I am now trying to get the thing to work using the HttpClient class in org.apache.*, but am running into a stupid "package org.apache..... does not exist", even though my CLASSPATH includes the directory where org is.

I posted that issue in a separate thread under the "New to Java" category, at
package org.apache.httpcomponents.httpclient.debian does not ex



Abdenour

[masijade]

You see those two script tags with the "src" param? Those are javascript files.

[achab]

You see those two script tags with the "src" param? Those are javascript files.

I see. I had looked for include and .js, but didn't think about looking for the src tag. Thanks for pointing that out. I will investigate it tomorrow (too tired today).

Abdenour

[achab]

Here is how JavaScript changes the web page from the one titled "Maricopa County Parcel Inquity" to a page titled "Maricopa County Tax History". The JavaScript function below doesn't give me the data I want, but goes farther than I managed to accomplish in Java. In Java, I have not able to get a web page titled "Maricopa County Tax History".

new WebForm_PostBackOptions(&quot;ctl00$cphMainContent $btnSubmit&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, false)

function WebForm_PostBackOptions(eventTarget, eventArgument, validation, validationGroup, actionUrl, trackFocus, clientSubmit) {
this.eventTarget = eventTarget;
this.eventArgument = eventArgument;
this.validation = validation;
this.validationGroup = validationGroup;
this.actionUrl = actionUrl;
this.trackFocus = trackFocus;
this.clientSubmit = clientSubmit;
}

The arguments passed to WebForm_PostBackOptions are:
actionURL = ""
clientSubmit = false
eventArgument = ""
eventTarget = "ctl00$cphMainContent$binSubmit"
trackFocus = false
validation = true
validationGroup = ""

I don't know much about JavaScript. Is the keyword new, coupled with ctl00$cphMainContent$binSubmit, what is creating the new page titled "Maricopa County Tax History" ? If so, and if all I wanted in Java was to get that far, how do I accomplish in Java the equivalent of what new/ctl00$cphMainContent$binSubmit did in JavaScript ?


Abdenour