Results 1 to 14 of 14
  1. #1
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Need help with WebClient

    Dear all,

    I am new member of this forum and also a beginner of JAVA so please forgive me is my question is not as accurate as it should be.

    Using WebClient, I want to connect to a URL which is running the open source ticketing system of OTRS. (more info on this at otrs.org)

    The issue I am facing is the following:

    - If I open the URL using any browser and going to the login page, the source of this page will contain some hidden fields. See bellow:

    XML Code:
    <form action="/otrs/index.pl" method="post" enctype="application/x-www-form-urlencoded" name="login"><input type="hidden" name="ChallengeToken" value="3b4fec379d97cf2a8931e3ed128d2249"/>
        <input type="hidden" name="Action" value="Login"/>
        <input type="hidden" name="RequestedURL" value="Action=Logout"/>
        <input type="hidden" name="Lang" value="en"/>
        <input type="hidden" name="TimeOffset" value=""/>
        <table border="0" cellspacing="0" cellpadding="3" width="280">
          <tr>
            <td align="center" class="singleboxhead">Welcome to xxxxxx</td>
          </tr>
          <tr>
            <td align="center" class="singleboxbody">
              <table cellspacing="8" cellpadding="2">
                <tr>
                  <td class="singleboxkey">User:</td>
                  <td class="singleboxvalue"><input type="text" name="User" value="" size="18"/></td>
                </tr>
                <tr>
                  <td class="singleboxkey">Password:</td>
                  <td class="singleboxvalue"><input type="password" name="Password" size="18"/></td>
                </tr>
              </table>
              <input class="button" type="submit" value="Login" onclick="return submit_form();"/>
            </td>
          </tr>
        </table>
      </form>
    - However, when I access the same URL using JAVA, the ChallengeToken hidden filed is missing from the source, while other hidden fields are there. So I think there is no issue with all hidden filed but one.

    XML Code:
      <form action="/otrs/index.pl" method="post" enctype="application/x-www-form-urlencoded" name="login">
        <input type="hidden" name="Action" value="Login"/>
        <input type="hidden" name="RequestedURL" value=""/>
        <input type="hidden" name="Lang" value="en"/>
        <input type="hidden" name="TimeOffset" value=""/>
        <table border="0" cellspacing="0" cellpadding="3" width="280">
          <tr>
            <td align="center" class="singleboxhead">Welcome to xxxxxx</td>
          </tr>
          <tr>
            <td align="center" class="singleboxbody">
              <table cellspacing="8" cellpadding="2">
                <tr>
                  <td class="singleboxkey">User:</td>
                  <td class="singleboxvalue"><input type="text" name="User" value="" size="18"/></td>
                </tr>
                <tr>
                  <td class="singleboxkey">Password:</td>
                  <td class="singleboxvalue"><input type="password" name="Password" size="18"/></td>
                </tr>
              </table>
              <input class="button" type="submit" value="Login" onclick="return submit_form();"/>
            </td>
          </tr>
        </table>
      </form>
    Because of the missing ChallengeToken field I cannot log into the site. Is there anyone have any idea why the ChallengeToken is missing from the source seen by Java?

    As a little debugging, I have tested my code with and without Browser Version (User-agent) but make no difference.

    Finally, this is the code I am using. Please note that my site using HTACCESS Authentication as well and this part have no issue. I am able to authenticate the HTACCESS prompt but the login form I am having issue with:


    Java Code:
    import java.io.IOException;
    import java.net.MalformedURLException;
    
    import org.apache.commons.codec.binary.Base64;
    
    import com.gargoylesoftware.htmlunit.BrowserVersion;
    import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
    import com.gargoylesoftware.htmlunit.WebClient;
    import com.gargoylesoftware.htmlunit.html.HtmlPage;
    
    
    public class OTRS {
    
    	public void LogIn() {
    		 
    		BrowserVersion ff = BrowserVersion.FIREFOX_24;
            BrowserVersion version = new BrowserVersion(ff.getApplicationName(), ff.getApplicationVersion(), "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16", ff.getBrowserVersionNumeric());
            WebClient webClient = new WebClient(version); 
            webClient.getOptions().setJavaScriptEnabled(false);
            webClient.getOptions().setRedirectEnabled(true);
            webClient.getOptions().setUseInsecureSSL(true);
            String userAndPassword = "user" + ":" + "pass";
    		String userAndPasswordBase64 = Base64.encodeBase64String(userAndPassword.getBytes());
    		webClient.addRequestHeader("Authorization", "Basic " + userAndPasswordBase64);
    		
            try {   
            	HtmlPage page = webClient.getPage("https://mydomain/otrs/index.pl");
            	System.out.println(page.getWebResponse().getContentAsString());
            	
            } catch (FailingHttpStatusCodeException e) {  
                e.printStackTrace();  
            } catch (MalformedURLException e){  
                e.printStackTrace();  
            } catch (IOException e) {  
                e.printStackTrace();  
            } 
    
    	}
    	
    }
    I would appreciate any hint/help on that one.

    I have coded the same kind of script before using perl with WWW::Mechanize module and I had no issue with it at all. Was working like it should. So due to this, I am confident that my server having no issue with this but somehow JAVA see things differently.

  2. #2
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,565
    Blog Entries
    7
    Rep Power
    21

    Default Re: Need help with WebClient

    The 'input' tag for the "ChallengeToken" is broken, i.e. it doesn't contain a '>' at the end of the tag. Was this your mistake or is it really missing? That could explain that the entire tag is missing ...

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  3. #3
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    Hello Jos,

    The input tag is correct however the forum was not able to display the complete line.

    The complete line is: after the form opening is:

    XML Code:
    <input type="hidden" name="ChallengeToken" value="3d86c1825a532c05bd41a933faafd69c"/>
    So teh above code is visible in a normal browser output but not in JAVA

  4. #4
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,040
    Rep Power
    20

    Default Re: Need help with WebClient

    Do you get a different result with Javascript enabled in WebClient?
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  5. #5
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    Quote Originally Posted by Tolls View Post
    Do you get a different result with Javascript enabled in WebClient?
    Hello,

    Javascript set to false or true does not change anything either.

    Java Code:
    webClient.getOptions().setJavaScriptEnabled(true);
    result

    XML Code:
    <form action="/otrs/index.pl" method="post" enctype="application/x-www-form-urlencoded" name="login">
        <input type="hidden" name="Action" value="Login"/>
        <input type="hidden" name="RequestedURL" value=""/>
        <input type="hidden" name="Lang" value="en"/>
        <input type="hidden" name="TimeOffset" value=""/>
        <table border="0" cellspacing="0" cellpadding="3" width="280">

  6. #6
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    Actually I have a strong feeling that ChallengeToken is simply not returned by the server when accessing using JAVA. It might be some protection mechanism against boots or something like that, however I am not able to prove this concept.

  7. #7
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,040
    Rep Power
    20

    Default Re: Need help with WebClient

    Use Fiddler to look at the communication with the server using the two different techniques (browser vs Java).
    If the response is different (as you say) then there will be some difference in that flow.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  8. #8
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    Quote Originally Posted by Tolls View Post
    Use Fiddler to look at the communication with the server using the two different techniques (browser vs Java).
    If the response is different (as you say) then there will be some difference in that flow.
    Ok, I have performed few test and the issue is not the "missing" filed.

    I have installed a test otrs server where I can play with and provide access to others to check this issue, but so far I have no idea why I cannot login.

    When I send the form, it simply return the login form without any error.

    The code I am using is:

    Java Code:
    import java.io.IOException;
    import java.net.MalformedURLException;
    
    //import org.apache.commons.codec.binary.Base64;
    
    import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
    import com.gargoylesoftware.htmlunit.ThreadedRefreshHandler;
    import com.gargoylesoftware.htmlunit.WebClient;
    import com.gargoylesoftware.htmlunit.html.HtmlForm;
    import com.gargoylesoftware.htmlunit.html.HtmlPage;
    import com.gargoylesoftware.htmlunit.html.HtmlPasswordInput;
    import com.gargoylesoftware.htmlunit.html.HtmlTextInput;
    
    public class OTRS {
    
    	public void LogIn(String username, String password ,String url) {
    		 
    		WebClient webClient = new WebClient(); 
    	    webClient.getOptions().setRedirectEnabled(true);
    	    webClient.getOptions().setJavaScriptEnabled(false);
    	    webClient.getOptions().setThrowExceptionOnScriptError(true); 
    	    webClient.setRefreshHandler(new ThreadedRefreshHandler()); 
    	    
    	    //String userAndPassword = username + ":" + password;
    		//String userAndPasswordBase64 = Base64.encodeBase64String(userAndPassword.getBytes());
    		//webClient.addRequestHeader("Authorization", "Basic " + userAndPasswordBase64);
    		
            webClient.getCookieManager().setCookiesEnabled(true);
            
            try{
            	webClient.getOptions().setUseInsecureSSL(true);
            	HtmlPage page = (HtmlPage) webClient.getPage(url);
                webClient.getOptions().setRedirectEnabled(true);
                
                HtmlForm form = page.getForms().get(0);
                HtmlTextInput usernametext = (HtmlTextInput) form.getInputByName("User");
                HtmlPasswordInput passwdtext = (HtmlPasswordInput) form.getInputByName("Password");
                usernametext.setValueAttribute(username);
                passwdtext.setValueAttribute(password);
           
        	    page = (HtmlPage) form.getInputByValue("Login").click();
                
                System.out.print(page.asText()); 
    
            } catch (FailingHttpStatusCodeException e) {
                e.printStackTrace();
            } catch (MalformedURLException e) {
                e.printStackTrace();
            } catch (IOException e) {
                e.printStackTrace();
            } 
    
    	}
    	
    }
    And I am calling it using the following code:

    Java Code:
    OTRS otrs = new OTRS();
    		otrs.LogIn("root@localhost","java","http://46.7.249.225/otrs/index.pl");
    If someone can help me out with this I would really say thank you. Feel free to use the login provided in the above code.

    This is just a test server so no harm can be done. It will be disabled as soon as there is a solution.

    Thanks in advance for whoever can help me.

  9. #9
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    Hmm, no matter what and how I try, it seems that JAVA simply cannot handle what I want to achieve. :( I guess I need to stack with Perl

  10. #10
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    I have found the issue, but still did not have the solution.

    Issue is with line:
    Java Code:
    page = (HtmlPage) form.getInputByValue("Login").click();
    The problem here is that the login page of OTRS does not use name for its submit button, so I wanted to use getInputByValue to find the button with Login written on it.

    However, the login form also have some hidden fields, and one of them having the value Login and that was the one found by getInputByValue, but that is not a submit button. So my script actually never submitted the form :) That is why the same login form was displayed over and over again.

    The only potential solution I see here is to modify the login form by adding a button to it on the fly so I can submit it.

  11. #11
    kneitzel is offline Senior Member
    Join Date
    Feb 2014
    Posts
    447
    Rep Power
    1

    Default Re: Need help with WebClient

    Hi,

    I just had a look at the api: did you try getButtonByName instead of getInputByName?
    And you could also call getInputsByName so you get all controls with that name.

    Just as an idea. I am just guessing.

    Konrad

  12. #12
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    Quote Originally Posted by kneitzel View Post
    Hi,

    I just had a look at the api: did you try getButtonByName instead of getInputByName?
    And you could also call getInputsByName so you get all controls with that name.

    Just as an idea. I am just guessing.

    Konrad
    Hello Kondrad,

    Yes, I have tested form.getButtonByName("Login") as well, however as stated earlier, the form submit button does not have name tag, only value. So this API would return ElementNotFoundException

    And unfortunately there is no API such form.getButtonByValue("Login") to get the button by its value.
    I was playing with HTMLElement to modify the form content so to add a submit button, and if I use that button to submit, the login works. I just found a bit frustrating that Java not able to handle this form correctly, or at least I do not know how :)

    So the challenge is to submit the following form:

    XML Code:
          <form action="/otrs/index.pl" method="post" enctype="application/x-www-form-urlencoded" name="login">
            <input type="hidden" name="Action" value="Login"/>
            <input type="hidden" name="RequestedURL" value=""/>
            <input type="hidden" name="Lang" value="en"/>
            <input type="hidden" name="TimeOffset" value=""/>
            <table border="0" cellspacing="0" cellpadding="3" width="280">
              <tbody>
                <tr>
                  <td align="center" class="singleboxhead">
                    Welcome to OTRS
                  </td>
                </tr>
                <tr>
                  <td align="center" class="singleboxbody">
                    <table cellspacing="8" cellpadding="2">
                      <tbody>
                        <tr>
                          <td class="singleboxkey">
                            User:
                          </td>
                          <td class="singleboxvalue">
                            <input type="text" name="User" value="root@localhost" size="18"/>
                          </td>
                        </tr>
                        <tr>
                          <td class="singleboxkey">
                            Password:
                          </td>
                          <td class="singleboxvalue">
                            <input type="password" name="Password" size="18" value="java"/>
                          </td>
                        </tr>
                      </tbody>
                    </table>
                    <input class="button" type="submit" value="Login" onclick="return submit_form();"/>
                  </td>
                </tr>
              </tbody>
            </table>
          </form>
    As you can see, java was able to enter user and password since we can see them in the HTML output of the form when it is redisplayed instead of submitted.

    I need to find a way to select and use the submit button:

    XML Code:
    <input class="button" type="submit" value="Login" onclick="return submit_form();"/>
    And finally, the following which should be able to select it does not work either:

    Java Code:
    form.getElementsByAttribute("submit", "value", "Login").get(0)

  13. #13
    rgabor is offline Member
    Join Date
    May 2014
    Posts
    9
    Rep Power
    0

    Default Re: Need help with WebClient

    And voila, the following code works, as stated earlier by adding a submit button on the fly to the current form.

    Java Code:
    import java.io.IOException;
    import java.net.MalformedURLException;
    
    import org.apache.commons.codec.binary.Base64;
    
    import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
    import com.gargoylesoftware.htmlunit.ThreadedRefreshHandler;
    import com.gargoylesoftware.htmlunit.WebClient;
    import com.gargoylesoftware.htmlunit.html.HtmlButton;
    import com.gargoylesoftware.htmlunit.html.HtmlForm;
    import com.gargoylesoftware.htmlunit.html.HtmlPage;
    import com.gargoylesoftware.htmlunit.html.HtmlPasswordInput;
    import com.gargoylesoftware.htmlunit.html.HtmlTextInput;
    
    public class OTRS {
    
    	public void LogIn(String username, String password ,String url) {
    		 
    		WebClient webClient = new WebClient(); 
    	    webClient.getOptions().setRedirectEnabled(true);
    	    webClient.getOptions().setJavaScriptEnabled(false);
    	    webClient.getOptions().setThrowExceptionOnScriptError(true); 
    	    webClient.setRefreshHandler(new ThreadedRefreshHandler()); 
    	    
    	    String userAndPassword = username + ":" + password;
    		String userAndPasswordBase64 = Base64.encodeBase64String(userAndPassword.getBytes());
    		webClient.addRequestHeader("Authorization", "Basic " + userAndPasswordBase64);
    		
            webClient.getCookieManager().setCookiesEnabled(true);
            
            try{
            	webClient.getOptions().setUseInsecureSSL(true);
            	HtmlPage page = (HtmlPage) webClient.getPage(url);
                webClient.getOptions().setRedirectEnabled(true);
                
                HtmlForm form = page.getForms().get(0);
                HtmlTextInput usernametext = (HtmlTextInput) form.getInputByName("User");
                HtmlPasswordInput passwdtext = (HtmlPasswordInput) form.getInputByName("Password");
                usernametext.setValueAttribute(username);
                passwdtext.setValueAttribute(password);
           
                // This is the solution I was able to come up with to submit the form
                // I am simply creating a submit button for the form I am working with
                HtmlButton submitButton = (HtmlButton)page.createElement("button");
                submitButton.setAttribute("type", "submit");
                form.appendChild(submitButton);
                page = submitButton.click();
    
                System.out.print(page.asXml());
                
                
            } catch (FailingHttpStatusCodeException e) {
                e.printStackTrace();
            } catch (MalformedURLException e) {
                e.printStackTrace();
            } catch (IOException e) {
                e.printStackTrace();
            } 
    
    	}
    	
    }
    Thanks everyone who try to help me.

  14. #14
    Robel Lakwe is offline Member
    Join Date
    Jun 2014
    Posts
    2
    Rep Power
    0

    Default Re: Need help with WebClient

    Robel Lakwe - thank you guys this help!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •