Page 1 of 2 12 LastLast
Results 1 to 20 of 39
  1. #1
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default Unable to write/post to a URL from Java

    Hi,

    I need to write 10101663 to the text field at treasurer.maricopa.gov/Parcel/Summary.aspx , hit Enter or Submit, then read the same URL after a few seconds.
    The problem is, I need to do it from Java code.

    I tried the code below, but it's not working. The data I read from treasurer.maricopa.gov/Parcel/Summary.aspx after trying to write 10101663 is the same data that was there all along, as if I didn't write anything.

    Below is one of the variants of several pieces of code that I tried, all unsuccessful. Could you tell me what's missing from my code ?

    import java.io.*;
    import java.util.*;
    import java.net.*;

    public class WebWriteRead {
    public static void main(String args[]) {
    try {
    URL taxIDURL = new URL("http://treasurer.maricopa.gov/Parcel/Summary.aspx");
    HttpURLConnection con = (HttpURLConnection) taxIDURL.openConnection();
    con.setDoInput (true);
    con.setDoOutput (true);
    con.setUseCaches (false);
    con.setRequestMethod("POST");
    con.setFollowRedirects(true);
    String msg = URLEncoder.encode("10101663");
    con.setRequestProperty("Content-Length", "" + msg.length());
    con.setRequestProperty("Content-Language", "en-US");
    con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
    DataOutputStream os = new DataOutputStream(con.getOutputStream());
    // OutputStreamWriter osw = new OutputStreamWriter(os);
    os.writeBytes(msg);
    os.flush();
    os.close();
    // any response?
    InputStreamReader isr = new InputStreamReader(con.getInputStream());
    BufferedReader br = new BufferedReader(isr);
    String line = null;
    while ( (line = br.readLine()) != null)
    System.out.println("line: " + line);


    }
    catch (MalformedURLException e) {
    System.out.println("Malformed URL: http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
    }
    catch (IOException e) {
    System.out.println("scanTreasurersWebSite - Caugh Exception " + e.toString());
    }
    }
    }
    Last edited by achab; 11-02-2010 at 07:14 AM.

  2. #2
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Open the url in "Post" mode, get the "OutputStream" and write a "param=value" pair, then close the OutputStream and open the "InputStream" to get the response. Examine the HTML source of the site to find out what "param=Value" pair(s) to pass.

  3. #3
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Quote Originally Posted by masijade View Post
    Open the url in "Post" mode, get the "OutputStream" and write a "param=value" pair, then close the OutputStream and open the "InputStream" to get the response. Examine the HTML source of the site to find out what "param=Value" pair(s) to pass.
    Hi Masijade,

    Thanks for the reply. How do I open the URL in post mode. Doesn't
    con.setDoOutput (true);
    accomplish that ? If not, which line of code should I change or add to open the URL in "Post" mode ?

    As far as "param=Value", the relevant section of the HTML source is:

    <input name="ctl00$cphMainContent$txtParcelNumber" type="text" maxlength="9" id="ctl00_cphMainContent_txtParcelNumber" style="width:90px;" />

    <input type="submit" name="ctl00$cphMainContent$btnSubmit" value="Submit" onclick="javascript:WebForm_DoPostBackWithOptions( new WebForm_PostBackOptions(&quot;ctl00$cphMainContent $btnSubmit&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, false))" id="ctl00_cphMainContent_btnSubmit" />

    Should I chose the name ctl00$cphMainContent$txtParcelNumber as param ? I tried both the name and the id yesterday, and neither worked. But something else may have been wrong with my code at that time too.

    Abdenour

  4. #4
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    The con.setRequestMethod("POST"); (among other things) does that, but you are not writing a param=value pair you are writing only the value.

    Look at what "javascript:WebForm_DoPostBackWithOptions" does. You may have "download" some included .js file to see it, but it is there and seemingly does some additional things.

  5. #5
    DarrylBurke's Avatar
    DarrylBurke is offline Member
    Join Date
    Sep 2008
    Location
    Madgaon, Goa, India
    Posts
    11,202
    Rep Power
    19

  6. #6
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    In answer to your reply on OTN that this was your only cross-post and your question as to whether DB means you should cross-post it more:

    No, he, and all others here, think you shouldn't crosspost at all, at least not without providing links to those threads and a clear statement that it is crossposted. Otherwise, everyone trying to help you is wasting their time. As clearly exemplified here, as I wasted my time giving you the same advice EJP had already given you.

  7. #7
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Quote Originally Posted by masijade View Post
    The con.setRequestMethod("POST"); (among other things) does that, but you are not writing a param=value pair you are writing only the value.

    Look at what "javascript:WebForm_DoPostBackWithOptions" does. You may have "download" some included .js file to see it, but it is there and seemingly does some additional things.
    Hi Masijade,

    I can't find an included .js file. I searched for the string include, and only found one instance (just regular text). I also only found one instance of .js, and it doesn't look at all like an include statement. It's inside a document.write() call.

    Abdenour

  8. #8
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Speaking of cross posting, I did post another technical question on Java-Forums.org at
    Could you tell me how to fix the offending 3 lines try block

    Technically, it's a different question. No write/post needed. Just reading a web page. From a business perspective, I only need one of the 2 questions answered. I wonder though if the problem in that thread could give you insight into what the problem is in this thread. In the other thread, I am getting an exception when I try to get a DataInputStream for
    treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663

    If you manually type treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 in your browser (other thread's issue), or manually write 10101663 in treasurer.maricopa.gov/Parcel/Summary.aspx (this thread's issue), you get the exact same result. Maybe my inabilities to do either from Java are related in some way.


    Abdenour
    Last edited by achab; 11-02-2010 at 05:30 PM.

  9. #9
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Quote Originally Posted by achab View Post
    In the other thread, I am getting an exception when I try to get a DataInputStream for
    treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663

    If you manually type treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 in your browser (other thread's issue), or manually write 10101663 in treasurer.maricopa.gov/Parcel/Summary.aspx (this thread's issue), you get the exact same result. Maybe my inabilities to do either from Java are related in some way.
    This may (probably) has more to do with the fact that the server attempts to determine if it is a browser doing the query or a program. Try setting the user agent (Google "Java user agent") to a typical browser user agent (google "internet explorer user agent").

    Edit: In which case you can use the "get" version. And what you need to use in the post version is, of course, "Parcel=10101663".

  10. #10
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Is there a way to spoof the User agent in Java so the web server thinks it's Firefox calling it ? I have seen the question asked in some forums, but not answered.

    Alternatively, in the "get" version, can my Java code have the Firefox on my machine read the web page, and then the Java code read the page source from my Firefox ? This alternative solution, if possible, might be unacceptably slow though. I eventually need to read over 10,000 of of such pages from my program and need that done in less than a 12 hours (and I have never done multi-thread programming before - so, for now at least, I plan to do that serially). So, if spoofing is possible, spoofing would be MUCH better. As would any solution where my Java successfully talks to the web server on the site from which I want to read.

  11. #11
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Did you google what I told you to? You can find plenty of examples with that. Also, you can do the "get" version with an HttpURLConnection as well, no need to try involving an external app. I'm sorry, but, jebus, just try it.

  12. #12
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Quote Originally Posted by masijade View Post
    Did you google what I told you to? You can find plenty of examples with that. Also, you can do the "get" version with an HttpURLConnection as well, no need to try involving an external app. I'm sorry, but, jebus, just try it.
    Hi Masijade,

    I did. Here is my new try block in the "get/read" method:

    try {
    // System.setProperty("http.agent", "");
    URL taxIDURL = new URL("http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
    HttpURLConnection conn = (HttpURLConnection) taxIDURL.openConnection();
    conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)");
    System.out.println ("conn.getResponsecode() = " + conn.getResponseCode());
    DataInputStream data = new DataInputStream(new BufferedInputStream(conn.getInputStream()));
    }

    Still getting the exception. Also, conn.getResponseCode() is 400.


    Jebus

  13. #13
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    400 is with authorisation. Do you have any sign in on that sight (maybe you use a Cookie). In any case google "Java HttpURLConnection authorisation". I'm sorry, but that is, currently, less a Java probelm (per se) than an understanding of HTTP. You first need to realise that it is about authorisation (and what that is and what it entails) before you can fix it and that requires an understanding of HTTP.

    Of course "the exception" doesn't help much, knowing what exception from exactly which line, might. The ResponseCode seems to tell the story, maybe.

  14. #14
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Quote Originally Posted by masijade View Post
    400 is with authorisation. Do you have any sign in on that sight (maybe you use a Cookie). In any case google "Java HttpURLConnection authorisation". I'm sorry, but that is, currently, less a Java probelm (per se) than an understanding of HTTP. You first need to realise that it is about authorisation (and what that is and what it entails) before you can fix it and that requires an understanding of HTTP.

    Of course "the exception" doesn't help much, knowing what exception from exactly which line, might. The ResponseCode seems to tell the story, maybe.
    The site doesn't require any sign in. See for yourself. Type treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 in your browser and you will see that no sign in is required. And, if this is the first time you go to that site, no cookie was needed.

    I use Firefox. It seems strange to me that Java code on my computer cannot get what Firefox can get from the same computer.

  15. #15
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Sorry 401 is authorisation 400 is bad request.

    And, BTW, Java has no problem with opening, reading, posting a web connection as long as it is done correctly.

    A little while back I had something I was writing and wanted to internationalise it a bit and wrote a little utility program that read my properties and connected to a dictionary website providing arguments for the source language, the target language, and the word and got back the translated word (wrote multiple lines for those that had multiple translations so that I had to choose which) all without a hitch. So, everything you want to do here is more than possible, and is usually easily done.

    You should probably be calling new HttpURLConnection and then all of those "set" statements and then call connect.

  16. #16
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    WebScanner.java:11: java.net.HttpURLConnection is abstract; cannot be instantiated

  17. #17
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Nevermind that. Just take this and see what you can get from it.

    Java Code:
    String item = URLEncoder.encode(tag, "UTF-8");
    
    Properties systemSettings = System.getProperties();
    systemSettings.put("proxySet", "true");
    systemSettings.put("http.proxyHost", "some.proxy.host");
    systemSettings.put("http.proxyPort", "80");
    
    URL url = new URL("http://www.freedict.com/onldict/onldict.php");
    sun.misc.BASE64Encoder encoder = new sun.misc.BASE64Encoder();  // there is now a better BASE64Encoder, BTW, I believe through the Preferences Classes
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    String encodedUserPwd = encoder.encode("domain\\user:xxxxxx".getBytes());
    connection.setRequestProperty("Proxy-Authorization", "Basic " + encodedUserPwd);
    connection.setRequestMethod("POST");
    connection.setDoOutput(true);
    OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
    try {
        out.write("search=" + item);
        out.write("&exact=true");
        out.write("&max=10");
        out.write("&from=English");
        out.write("&to=" + lang);
        out.write("&fname=" + fname);
        out.write("&back=" + back);
    } finally { try { out.close(); } catch (Exception e) {} };
    BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
    try {
        String decodedString;
        while ((decodedString = in.readLine()) != null) {
            if (decodedString.contains("result-r-blue")) {
                decodedString = decodedString.replaceFirst("^.*<(em|strong)>", "").replaceFirst("</(em|strong)>.*$", "");
                if (!decodedString.contains("No matches found")) {
                    value = decodedString;
                }
            }
        }
    } finally { try { in.close(); } catch (Exception e) {} }
    Last edited by masijade; 11-03-2010 at 08:04 AM. Reason: reformatted code abit

  18. #18
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    Quote Originally Posted by achab View Post
    WebScanner.java:11: java.net.HttpURLConnection is abstract; cannot be instantiated
    I know, it had been a while though. I was already reposting when you posted.

  19. #19
    masijade is offline Senior Member
    Join Date
    Jun 2008
    Posts
    2,571
    Rep Power
    9

    Default

    P.S. that entire block of code should be wrapped in a try catch I simply didn't include that try, nor the catch block.

  20. #20
    achab is offline Member
    Join Date
    Nov 2010
    Posts
    26
    Rep Power
    0

    Default

    Thanks Masijade for sharing your code. Part of it is related to writing/posting, which I ignored. I copied what might be related to reading. My new try block (listed below) still fails with the same error message. Here is one things that puzzles me. I ask the web server to open treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663 , and, from the stacktrace, the web server thinks I am trying to open treasurer.maricopa.gov/Parcel/SetSession.asp?pn=10101663&tcd=1&fpn=101-01-663

    code:
    try {
    // System.setProperty("http.agent", "");
    // String item = URLEncoder.encode(tag, "UTF-8");

    Properties systemSettings = System.getProperties();
    systemSettings.put("proxySet", "true");
    systemSettings.put("http.proxyHost", "some.proxy.host");
    systemSettings.put("http.proxyPort", "80");

    CookieHandler.setDefault(new CookieManager());
    URL taxIDURL = new URL("http://treasurer.maricopa.gov/parcels/default.asp?Parcel=10101663");
    HttpURLConnection conn = (HttpURLConnection) taxIDURL.openConnection();
    conn.setRequestProperty("Proxy-Authorization", "Basic ");
    conn.setRequestProperty("User-Agent", "MSIE");
    conn.setFollowRedirects(true);
    System.out.println ("conn.getResponsecode() = " + conn.getResponseCode());
    BufferedReader data = new BufferedReader(new InputStreamReader(conn.getInputStream()));
    }

Page 1 of 2 12 LastLast

Similar Threads

  1. Unable to create and write files
    By DrKilljoy in forum New To Java
    Replies: 4
    Last Post: 09-05-2010, 12:55 AM
  2. Replies: 8
    Last Post: 04-30-2010, 10:25 AM
  3. HTTPS POST Using Java
    By drcman in forum Advanced Java
    Replies: 7
    Last Post: 02-13-2010, 02:19 PM
  4. Replies: 3
    Last Post: 05-04-2009, 03:15 AM
  5. Post Method in java.net
    By freddieMaize in forum Advanced Java
    Replies: 2
    Last Post: 02-23-2009, 02:59 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •