Page 2 of 2 FirstFirst 12
Results 21 to 39 of 39
  1. #21
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    It seems to be using some homegrown sessions and looking at some of the HTML source there are a few hidden fields that are passed back and forth and those probably control the sessions. Look at the source again (and that one javascript method is contained in that page) and see what other fields you might be able to pass along. The "mechanics" of this seem to be correct, it seems, now, as though you simply need to find the right information to post. If need be get a packet sniffer and make the connection with the browser and then examine the network traffic that resulted and see what is really going on.

  2. #22
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    For now, I have given up trying to post anything to the site ... only trying to read. Do you still think the source and included Javascript will help when all I am trying to do is read the HTML page that the web site is creating ?

  3. #23
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Whether with "Get" or "Post" you are still providing information. And that site seems to run through a couple of redirects (building up a session as it goes, seemingly). I still think your best bet would be to go to the site where you have to enter the number, then start a packet sniffer and then enter a number submit the form and find out exactly what it is that the form submits and recreate (whether with get or post probably won't matter).

  4. #24
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Thanks for the tip. I will try that tomorrow. My brain batteries run out for tonight.

  5. #25
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Here is the first HTTP line from WireShark. There are other lines, but let's start with the first:
    Protocol = HTTP
    Info = GET /Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S HTTP/1.1

    Based on the above, I added the following to my code:
    String msg = "pn=" + URLEncoder.encode("10101663");
    msg += "&tcd=" + URLEncoder.encode("1");
    msg += "&fpn=" + URLEncoder.encode("101-01-663%201");
    msg += "&mry=" + URLEncoder.encode("2010");
    msg += "&sec=" + URLEncoder.encode("True");
    msg += "&rf=" + URLEncoder.encode("False");
    msg += "&dp=" + URLEncoder.encode("S");

    msg is what I ultimately send to the web server.

    How would you translate the "/Parcel/SetSession.asp?" part into Java code ?

    It takes a few seconds for the web server to come back with the information I need. Should I make my program sleep a few seconds, or is there a smarter way to detect when the web server is ready to serve me the information I need ?

    Abdenour
    Last edited by achab; 11-04-2010 at 09:09 AM.

  6. #26
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    http://<server>/Parcel/SetSession.asp

    With the rest of that as the part written to the outputstream in the post call.

    As far as sleeping, no, simply attempt to open the inputstream and it will "wait" until it gets some data, or until the server closes it.

    But this "rf=False$dp=S" is
    Java Code:
    msg += "&rf=" + URLEncoder.encode("False$dp=S");
    and not
    Java Code:
    msg += "&rf=" + URLEncoder.encode("False");
    msg += "&dp=" + URLEncoder.encode("S");
    if that is really a "$" and not an "&".

    BTW, that String is already URL encoded (%20 is " "), so there is no reason to call encode at all, and if you do change
    Java Code:
    msg += "&fpn=" + URLEncoder.encode("101-01-663%201");
    to
    Java Code:
    msg += "&fpn=" + URLEncoder.encode("101-01-663 1");

  7. #27
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Or, if you want to use a get simply use
    Java Code:
    "http://treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S"
    as the url

  8. #28
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Oops ... it's rf=False&dp=S
    I changed 101-01-663%201 to 101-01-663 1 in the write/post version.

    With the new URL, the "get/read" version of the code no longer throws an exception. But, both the "get/read" and the "write/post followed by read" versions of the code give me an HTTP page source that contains "Maricopa County Parcel Inquiry", which is the page source you get if you go to treasurer.maricopa.gov/Parcel/Default.aspx and don't type anything.

    As for the input stream waiting until it gets some data, when you do it manually, there is data there before you type anything. It just happens that the data I want is the one after you type, hit Submit, and wait a few seconds. Making the code sleep 5 seconds before getting in the InputStream doesn't solve the problem though.

    Abdenour

  9. #29
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Here is the new full version of the "Get/Read" version of the code, which still doesn't work.
    Java Code:
    import java.io.*;
    import java.util.*;
    import java.net.*;
    
    public class WebScanner {
        public static void main(String args[]) {
            try {
    //            URL taxIDURL = new URL("http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
                URL taxIDURL = new URL("http://treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S");
                HttpURLConnection conn = (HttpURLConnection) taxIDURL.openConnection();
                conn.setRequestProperty("Proxy-Authorization", "Basic ");
                conn.setRequestProperty("User-Agent", "MSIE");
                conn.setFollowRedirects(true);
                BufferedReader data = new BufferedReader(new InputStreamReader(conn.getInputStream()));
                for ( int i=0 ; i<45 ; i++ ) {
                    String line = data.readLine();
                    System.out.println ("line = " + line);
                }
            }
            catch (MalformedURLException e) {
                System.out.println("Malformed URL: http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
            }
            catch (IOException e) {
                System.out.println("scanTreasurersWebSite - Caugh Exception " + e.toString());
                e.printStackTrace();
            }
        }
    }
    Last edited by achab; 11-04-2010 at 09:52 AM.

  10. #30
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    push "QUOTE" on one of my posts with code tags and you will see them.

  11. #31
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Thanks. I just did it. So the progress in the "Get/Read" version from yesterday is that there is no longer an exception thrown. The progress that needs to be made is to get the same HTTP one would get manually from typing:
    treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S
    into a web browser and waiting 1 second or so.

  12. #32
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    When I try it I get a "server redirected too many times" error. The server seems to be actively attempting to block automated lookups. I have the feeling the problem is at the server side.

    BTW
    Java Code:
    conn.setFollowRedirects(true);
    should be
    Java Code:
    HttpURLConnection.setFollowRedirects(true);
    and should be done before calling openConnection.

  13. #33
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Thanks Masijade. I changed that. The problem still remains though. The output is the same: the page source of what you would get if you went to treasurer.maricopa.gov/Parcel/Default.aspx

    I just rerun WireShark, this time simulating the "Get/Read" version, that is, hitting enter on my browser's URL, in which I had typed treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S before starting sniffing.

    Here is the Info for first HTTP and HTTP/XML lines in WireShark. I highlighted in Bold the few lines that I think are relevant:

    GET /root.sxml HTTP/1.1
    HTTP/1.1 200 OK
    HTTP/1.1 200 OK
    HTTP/1.1 200 OK
    GET /WANIPConn1.xml HTTP/1.1
    HTTP/1.1 200 OK
    POST /sqm/windowsLive/sqmserver.dll HTTP/1.1
    HTTP /1.1 403 .6 Forbidden (text/html)
    GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship
    HTTP/1.1 301 Moved Permanently
    GET /capacity HTTP/1.1
    HTTP/1.1 200 OK (text/plain)
    GET /capacity HTTP/1.1
    HTTP/1.1 200 OK (text/plain)
    GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship/~ts-101104/~l-en/config.xml HTTP/1.1
    HTTP/1.1 200 OK
    GET /Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S HTTP/1.1
    HTTP/1.1 302 Object moved (text/html)
    GET /Parcel/Summary.aspx HTTP/1.1
    GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship
    HTTP/1.1 200 OK (text/plain)
    GET /1reupdate/short?!/~Live.ConfigServer.SuiteUpdate/~/~/~/~/~op-GetShortCatalog-ship/~ts-101104/~l-en/config.xml HTTP/1.1
    HTTP/1.1 200 OK
    GET /_utm.gif?utmwv=4.8.6&utmn=1516772251&utmhn=treasur er.maricopa.gov&utmcs=UTF-8&utmsr=1280x800&utmcs=32-bit&utmul=en-us&utmje=1&utmfl=..


    I can't read the full last line above. How would you translate it into code ?

  14. #34
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Quote Originally Posted by masijade View Post
    The con.setRequestMethod("POST"); (among other things) does that, but you are not writing a param=value pair you are writing only the value.

    Look at what "javascript:WebForm_DoPostBackWithOptions" does. You may have "download" some included .js file to see it, but it is there and seemingly does some additional things.
    It seems like javascript:WebForm_DoPostBackWithOptions is a standard JavaScript method/function, not something defined in an included .js file. Is there an easy way to integrate Java with JavaScript, so that my code calls javascript:WebForm_DoPostBackWithOptions ?

    Abdenour

  15. #35
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    No, it's not a standard. If you look at the source of the page where you enter the number it is contained at the top of the page between SCRIPT tags.

  16. #36
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Quote Originally Posted by masijade View Post
    Or, if you want to use a get simply use
    Java Code:
    "http://treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663%201&mry=2010&sec=True&rf=False&dp=S"
    as the url
    Quote Originally Posted by masijade View Post
    No, it's not a standard. If you look at the source of the page where you enter the number it is contained at the top of the page between SCRIPT tags.
    I have done a search through Ctrl-F, and don't see it. Here are the script tags:

    Java Code:
    <script type="text/javascript">
    //<![CDATA[
    var theForm = document.forms['aspnetForm'];
    if (!theForm) {
        theForm = document.aspnetForm;
    }
    function __doPostBack(eventTarget, eventArgument) {
        if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
            theForm.__EVENTTARGET.value = eventTarget;
            theForm.__EVENTARGUMENT.value = eventArgument;
            theForm.submit();
        }
    }
    //]]>
    </script>
    .......
    Java Code:
    <script src="/Parcel/WebResource.axd?d=82X_DmeRwShoLJobNHsQnbVOlScs8_yWxC9DCuLI2YS__W5fmnbySsuN6RVESgJ05_dGX5bZ3fQBxx2amMPSDiePiBQ1&amp;t=634234388291823091" type="text/javascript"></script>
    .................
    Java Code:
    <script src="/Parcel/WebResource.axd?d=0lTb8rR5d7hmnuOluNU1bbXvkHvoFQ_jnwji6bwN6BWAUT5z66CJqXrS0vVoz3RIP0OL323hu9vmTevPzdVyoW8A-7U1&amp;t=634234388291823091" type="text/javascript"></script>
    ................
    Java Code:
    <script type="text/javascript">
    //<![CDATA[
    WebForm_AutoFocus('ctl00_cphMainContent_txtParcelNumber');//]]>
    </script>
    </form>
     <script type="text/javascript">
    	var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
    	document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
    </script>
    <script type="text/javascript">
    	var pageTracker = _gat._getTracker("UA-2900680-1");
    	pageTracker._initData();
    	pageTracker._trackPageview();
    </script>

    I am now trying to get the thing to work using the HttpClient class in org.apache.*, but am running into a stupid "package org.apache..... does not exist", even though my CLASSPATH includes the directory where org is.

    I posted that issue in a separate thread under the "New to Java" category, at
    package org.apache.httpcomponents.httpclient.debian does not ex



    Abdenour

  17. #37
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    You see those two script tags with the "src" param? Those are javascript files.

  18. #38
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Quote Originally Posted by masijade View Post
    You see those two script tags with the "src" param? Those are javascript files.
    I see. I had looked for include and .js, but didn't think about looking for the src tag. Thanks for pointing that out. I will investigate it tomorrow (too tired today).

    Abdenour

  19. #39
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Here is how JavaScript changes the web page from the one titled "Maricopa County Parcel Inquity" to a page titled "Maricopa County Tax History". The JavaScript function below doesn't give me the data I want, but goes farther than I managed to accomplish in Java. In Java, I have not able to get a web page titled "Maricopa County Tax History".

    new WebForm_PostBackOptions(&quot;ctl00$cphMainContent $btnSubmit&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, false)

    Java Code:
    function WebForm_PostBackOptions(eventTarget, eventArgument, validation, validationGroup, actionUrl, trackFocus, clientSubmit) {
    this.eventTarget = eventTarget;
    this.eventArgument = eventArgument;
    this.validation = validation;
    this.validationGroup = validationGroup;
    this.actionUrl = actionUrl;
    this.trackFocus = trackFocus;
    this.clientSubmit = clientSubmit;
    }
    The arguments passed to WebForm_PostBackOptions are:
    actionURL = ""
    clientSubmit = false
    eventArgument = ""
    eventTarget = "ctl00$cphMainContent$binSubmit"
    trackFocus = false
    validation = true
    validationGroup = ""

    I don't know much about JavaScript. Is the keyword new, coupled with ctl00$cphMainContent$binSubmit, what is creating the new page titled "Maricopa County Tax History" ? If so, and if all I wanted in Java was to get that far, how do I accomplish in Java the equivalent of what new/ctl00$cphMainContent$binSubmit did in JavaScript ?


    Abdenour

Page 2 of 2 FirstFirst 12

Similar Threads

  1. Unable to create and write files
    By DrKilljoy in forum New To Java
    Replies: 4
    Last Post: 09-05-2010, 01:55 AM
  2. Replies: 8
    Last Post: 04-30-2010, 11:25 AM
  3. HTTPS POST Using Java
    By drcman in forum Advanced Java
    Replies: 7
    Last Post: 02-13-2010, 03:19 PM
  4. Replies: 3
    Last Post: 05-04-2009, 04:15 AM
  5. Post Method in java.net
    By freddieMaize in forum Advanced Java
    Replies: 2
    Last Post: 02-23-2009, 03:59 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •