Trying to login to a website then grab another page

[DaveTheAve]

Alright so I'm ripping my hair out searching the heck out of google trying to find this out and I'm thinking I'm the ONLY one in the world who has ever tried to do this. lol

I'm trying to login to a website with Java then go to another page on the same site while logged in and grab the HTML off that site to parse every set period of time (like every 3min.)

The code i've been betting like a dead horse is this:

/**
 *  $Id: LoginByHttpPost.java 101 2010-03-13 23:52:31Z oneyour $
 * 
 * This is an accompanying program for the article
 * <a href="http://www.1your.com/drupal/LoginToWebsiteByHTTPPOSTCodeListing
" title="http://www.1your.com/drupal/LoginToWebsiteByHTTPPOSTCodeListing
">http://www.1your.com/drupal/LoginToWebsiteByHTTPPOSTCodeListing
</a> * 
 * Copyright (c) 2009 - 2010 <a href="http://www.1your.com" title="www.1your.com">www.1your.com</a>.  All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of <a href="http://www.1your.com" title="www.1your.com">www.1your.com</a> nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 * 
 */
 
import java.net.*;
import java.io.*;
 
public class LoginByHttpPost
{
    private static final String POST_CONTENT_TYPE = "application/x-www-form-urlencoded";
    private static final String LOGIN_ACTION_NAME = "login";
    private static final String LOGIN_USER_NAME_PARAMETER_NAME = "guideHandle";
    private static final String LOGIN_PASSWORD_PARAMETER_NAME = "password";
    private static final String LOGIN_URL = "https://universe.chacha.com/login";
 
    private static final String LOGIN_USER_NAME = "Kitler13";
    private static final String LOGIN_PASSWORD = "Big91l0w";
    
    private static final String TARGET_URL = "https://universe.chacha.com/login";
 
    /**
     * The single public method of this class that
     * 1. Prepares a login message
     * 2. Makes the HTTP POST to the target URL
     * 3. Reads and returns the response
     *
     * @throws IOException
     * Any problems while doing the above.
     *
     */
    public void httpPostLogin ()
    {
        try
        {
            // Prepare the content to be written
            // throws UnsupportedEncodingException
            String urlEncodedContent = preparePostContent(LOGIN_USER_NAME, LOGIN_PASSWORD);
 
            HttpURLConnection urlConnection = doHttpPost(LOGIN_URL, urlEncodedContent);
            
            String response = readResponse(urlConnection);
            System.out.println("Recevied response is: '/n" + response + "'");
         }
        catch(IOException ioException)
        {
            System.out.println("Problems encounterd.");
        }
    }
 
    /**
     * Using the given username and password, and using the static string variables, prepare
     * the login message. Note that the username and password will encoded to the
     * UTF-8 standard.
     *
     * @param loginUserName
     * The user name for login
     *
     * @param loginPassword
     * The password for login
     *
     * @return
     * The complete login message that can be HTTP Posted
     *
     * @throws UnsupportedEncodingException
     * Any problems during URL encoding
     */
    private String preparePostContent(String loginUserName, String loginPassword) throws UnsupportedEncodingException
    {
        // Encode the user name and password to UTF-8 encoding standard
        // throws UnsupportedEncodingException
        String encodedLoginUserName = URLEncoder.encode(loginUserName, "UTF-8");
        String encodedLoginPassword = URLEncoder.encode(loginPassword, "UTF-8");
 
        String content = "login=" + LOGIN_ACTION_NAME +" &" + LOGIN_USER_NAME_PARAMETER_NAME +"="
        + encodedLoginUserName + "&" + LOGIN_PASSWORD_PARAMETER_NAME + "=" + encodedLoginPassword;
 
        return content;
 
    }
 
    /**
     * Makes a HTTP POST to the target URL by using an HttpURLConnection.
     *
     * @param targetUrl
     * The URL to which the HTTP POST is made.
     *
     * @param content
     * The contents which will be POSTed to the target URL.
     *
     * @return
     * The open URLConnection which can be used to read any response.
     *
     * @throws IOException
     */
    public HttpURLConnection doHttpPost(String targetUrl, String content) throws IOException
    {
        HttpURLConnection urlConnection = null;
        DataOutputStream dataOutputStream = null;
        try
        {
            // Open a connection to the target URL
            // throws IOException
            urlConnection = (HttpURLConnection)(new URL(targetUrl).openConnection());
 
            // Specifying that we intend to use this connection for input
            urlConnection.setDoInput(true);
 
            // Specifying that we intend to use this connection for output
            urlConnection.setDoOutput(true);
 
            // Specifying the content type of our post
            urlConnection.setRequestProperty("Content-Type", POST_CONTENT_TYPE);
 
            // Specifying the method of HTTP request which is POST
            // throws ProtocolException
            urlConnection.setRequestMethod("POST");
            
            // Prepare an output stream for writing data to the HTTP connection
            // throws IOException
            dataOutputStream = new DataOutputStream(urlConnection.getOutputStream());
 
            // throws IOException
            dataOutputStream.writeBytes(content);
            dataOutputStream.flush();
            dataOutputStream.close();
 
            return urlConnection;
        }
        catch(IOException ioException)
        {
            System.out.println("I/O problems while trying to do a HTTP post.");
            ioException.printStackTrace();
 
            // Good practice: clean up the connections and streams
            // to free up any resources if possible
            if (dataOutputStream != null)
            {
                try
                {
                    dataOutputStream.close();
                }
                catch(Throwable ignore)
                {
                    // Cannot do anything about problems while
                    // trying to clean up. Just ignore
                }
            }
            if (urlConnection != null)
            {
                urlConnection.disconnect();
            }
 
            // throw the exception so that the caller is aware that
            // there was some problems
            throw ioException;
        }
    }
 
    /**
     * Read response from the URL connection
     *
     * @param urlConnection
     * The URLConncetion from which the response will be read
     *
     * @return
     * The response read from the URLConnection
     *
     * @throws IOException
     * When problems encountered during reading the response from the
     * URLConnection.
     */
    private String readResponse(HttpURLConnection urlConnection) throws IOException
    {
 
        BufferedReader bufferedReader = null;
        try
        {
            // Prepare a reader to read the response from the URLConnection
            // throws IOException
            bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
            String responeLine;
 
            // Good Practice: Use StringBuilder in this case
            StringBuilder response = new StringBuilder();
 
            // Read until there is nothing left in the stream
            // throws IOException
            while ((responeLine = bufferedReader.readLine()) != null)
            {
                response.append(responeLine + "\n");
            }
 
            return response.toString();
        }
        catch(IOException ioException)
        {
            System.out.println("Problems while reading the response");
            ioException.printStackTrace();
 
            // throw the exception so that the caller is aware that
            // there was some problems
            throw ioException;
 
        }
        finally
        {
            // Good practice: clean up the connections and streams
            // to free up any resources if possible
            if (bufferedReader != null)
            {
                try
                {
                    // throws IOException
                    bufferedReader.close();
                }
                catch(Throwable ignore)
                {
                    // Cannot do much with exceptions doing clean up
                    // Ignoring all exceptions
                }
            }
 
        }
    }
}

Can you please help me you wonderful Java Gods?

:D Pretty Please? :p

[iluxa]

what's the problems you're experiencing? exceptions? empty input stream?

[DaveTheAve]

It doesn't hold the cookies and I can not figure out to change the URL while still effectively being signed in.

[iluxa]

sounds like you will have to deal with the cookies, since the website you're accessing decides whether to let you in based on these cookies. take a look here:

HOW-TO: Handling Cookies Using the java.net API

doesn't seem too complex

[DaveTheAve]

That actually MUCH simpler than what I was doing thanks!

Now how would I change the URL once I'm signed in to be able to parse the passworded content?

[iluxa]

i think all you need to do is open a new HttpUrlConnection to the new URL, and make sure to give it the cookies you've received

[DaveTheAve]

Can someone please confirm all I'd need to do is:

public static void main(String[] args) {
CookieManager cm = new CookieManager();
try {
URL url = new URL("http://www.hccp.org/test/cookieTest.jsp");
URLConnection conn = url.openConnection();
conn.connect();
cm.storeCookies(conn);
System.out.println(cm);

URL url = new URL("http://www.hccp.org/member/dashboard.php");

cm.setCookies(url.openConnection());
} catch (IOException ioe) {
ioe.printStackTrace();
}
}

??