How do you get and set cookies with HttpURLConnection

[J-Live]

Can someone explain the steps involved? I can't seem to follow the tutorials I've found on the web. Do you have to connect to the same URL twice in order to send the cookie back to the server? This is the only example I could find that worked for getting the cookies.

 String headerfields = conn.getHeaderField(0);
    System.out.println(headerfields);
    System.out.println("---Start of headers---");
    int i = 1;
    while ((headerfields = conn.getHeaderField(i)) != null) {
    String key = conn.getHeaderFieldKey(i);
    System.out.println(((key==null) ? "" : key + ": ") + headerfields);
    i++;
    
    }
     cookie = conn.getHeaderField("Set-Cookie");
     if (cookie != null)
		cookie = cookie.substring(0, cookie.indexOf(';'));
	        System.out.println("cookie: " + cookie);

:( Can someone please give me a simple example of how to get and set cookies from a HttpURLConnection.

[fishtoprecords]

you set a cookie when you write a response, not when you handle the input.

Exactly how depends on how you are creating the response.

[J-Live]

Yes, that's part of my question. I tried to open a new httpurlconnection and send the cookie like conn2.setRequestProperty("Cookie", cookie); but that brings response code errors. So, after I get the cookie do I have to disconnect and make a new connection? I'm basically asking, what do you do after you get the cookie. What's the next step. I'm trying to download files from servers but some of them will require cookies to be returned, I assume.

[Norm]

that brings response code errors

Can you copy and paste the error message here?

Cookies are initially set by the server and sent to a client.
The client saves the cookie locally.
When the client connects to that server at a later time, the client sends (returns) the cookie that the server sent it.

Your code looks like it would get the cookie sent by the server.

Are you sure that the cookie you are returning to the server is the one that the server send to you originally? Can there be other things the server is looking at besides the cookie to cause it to generate the Status/response code you are getting?

[J-Live]

Can you copy and paste the error message here?

java.io.IOException: Server returned HTTP response code: 503 for



I get that one or a 404 response code error.


Cookies are initially set by the server and sent to a client.
The client saves the cookie locally. I thought that was what my cookie variable was doing. It always prints null though.
When the client connects to that server at a later time, the client sends (returns) the cookie that the server sent it. So, you do have to make two separate connections to the URL?

Your code looks like it would get the cookie sent by the server.

Are you sure that the cookie you are returning to the server is the one that the server send to you originally? Can there be other things the server is looking at besides the cookie to cause it to generate the Status/response code you are getting?

The server could possibly be looking at other things. I'm not sure. I think for rapid-share the server is redirecting to another link or something like that, that site always returns a 404 even before I tried to add cookies to my code. All the other sites I've tried were working before I tried to add the cookie code. That's why I'm wondering if I'm supposed to disconnect then make a new connection or not. I'll post the relevant code and pseudocode I'm trying to use, maybe that'll help.

start of while loop with counter initialized to 0
if counter is 0
URI decoder = new URI(urlstring);
url = decoder.toURL();
dlfilename = url.getFile();
String holdername = dlfilename.substring(dlfilename.lastIndexOf('/') + 1);
fname = holdername.substring(0, holdername.lastIndexOf('.') + 4).replaceAll("\\p{Punct}&&[./]", "");
[B]HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; H010818)" );
conn.setUseCaches(false);
conn.setFollowRedirects(false);
conn.setAllowUserInteraction(true);
conn.setDoInput(true);
[/B]

[B]int responseCode = conn.getResponseCode();
String responsemessage = conn.getResponseMessage();
System.out.println("Response code from server=" + responseCode + responsemessage);
			if (responseCode  / 100 != 2)
			{
			System.out.println("An error has occured check the response code for details" + conn.getErrorStream());
			}//end get response code
conn.connect(); 
 in = conn.getInputStream();
 String headerfields = conn.getHeaderField(0);
    System.out.println(headerfields);
    System.out.println("---Start of headers---");
    int i = 1;
    while ((headerfields = conn.getHeaderField(i)) != null) {
    String key = conn.getHeaderFieldKey(i);
    System.out.println(((key==null) ? "" : key + ": ") + headerfields);
    i++;
    
    }
     cookie = conn.getHeaderField("Set-Cookie");
     if (cookie != null)
		cookie = cookie.substring(0, cookie.indexOf(';'));
	        System.out.println("cookie: " + cookie);
in.close();
conn.disconnect();
end if counter is 0

if counter is 1

 URI decoder2 = new URI(urlstring);[/B]
     url = decoder2.toURL();
     dlfilename2 = url.getFile();
String holdername2 = dlfilename2.substring(dlfilename2.lastIndexOf('/') + 1);
fname = holdername2.substring(0, holdername2.lastIndexOf('.') + 4).replaceAll("\\p{Punct}&&[./]", "");
[B]     conn2 = (HttpURLConnection)url.openConnection();
     conn2.setRequestMethod("POST");
     conn2.setDoOutput(true);
     conn2.setRequestProperty("Cookie", cookie); 
     System.out.println("Sent Cookie" + cookie);
     conn2.setRequestMethod("GET");
conn2.addRequestProperty("Range", "bytes=" + footerstart + "-");

 int responseCode2 = conn2.getResponseCode();
String responsemessage2 = conn2.getResponseMessage();
System.out.println("Response code from server=" + responseCode2 + responsemessage2);
			if (responseCode2  / 100 != 2)
			{
			System.out.println("An error has occured check the response code for details" + conn.getErrorStream());
			}//end get response code
conn2.connect(); 
                    in2 = conn2.getInputStream();
read from inputstream and write data to file
end if counter is 1 

add 1 to counter 
end while loop[/B]

[Norm]

Status code 404 - server can't find requested resource
That might not have anything to do with cookies. Does the request exist on the server?

It always prints null though.

If there isn't a Set-Cookie line in the response from the server, then there is no cookie to be returned.

[J-Live]

Yes, the resource exists, but like I said, the server may be redirecting maybe to another location on the server where the file is actually located at. I don't know much about servers.

This is the output I get when the 503 error occurs.

HTTP/1.1 200 OK
---Start of headers---
Server: nginx/0.6.32
Date: Fri, 07 Nov 2008 20:40:45 GMT
Content-Type: application/octet-stream
Content-Length: 119159508
Last-Modified: Tue, 23 Sep 2008 03:55:12 GMT
Connection: keep-alive
Content-Disposition: attachment
Accept-Ranges: bytes
ETag: ""
Accept-Ranges: bytes
cookie: null
Sent Cookienull

Here's what I get with the 404 response.
HTTP/1.1 200 OK
---Start of headers---
Date: Fri, 07 Nov 2008 21:43:47 GMT
Connection: close
Content-Type: application/octet-stream
Accept-Ranges: bytes
Content-Disposition: Attachment; filename=ACS3EXT_by_605.rar
Content-Length: 50246931
cookie: null
Sent Cookienull

Here's the error code that is given when the server returns a 404 response.

java.io.FileNotFoundException: htt-p://rs156cg2.rapidshare.co-m/files/57307159/1465247/ACS3EXT_by_605.rar?mirror=on&x=70&y=42

[Norm]

This is the output I get when the 503 error occurs.
HTTP/1.1 200 OK

You have some problem with what is happening and what you are reporting. The response header you've posted here shows a Status/response code of 200 not 503???

How/where are you getting the responses you've posted?

Here is what my server gets from my browser when I sent it an HTTP GET:

GET / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.13) Gecko/20060414
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: wowbb=%7C%7C%7C%7C-300

Here's what my server sends back to the browser:

HTTP/1.0 200 OK
Date: Nov 7, 2008
Server: NormsHTTPServer/0.1
Set-Cookie: LastVisit=7 Nov 2008 22:52:08 GMT; expires=Wednesday, 31-Dec-2003 23:59:59 GMT;
Last-Modified: Sun, 2 Oct 2005 12:48:22 GMT
Content-Type: text/html
Content-Length: 4583

If my simple HTTP server would be of any use to you, it can be downloaded from:
All files on Norm's Google site
Look at the HTTPFromJarW.bat batch file for the files needed plus all that start with HTTPServer. I think they are all on the site.
Also there are several environment variables used whose values could be replaced with values:
DEV_DRIVE=D:
DEV_HOME=D:
DOCS_HOME=C:

[J-Live]

I should have said before the errors occur. I was using two connections. I opened one then read the cookie. That's where the output came from. I closed that connection. Then I opened a new HttpURLConnection to the same url and tried to post the cookie. That's when I get the error. I thought about what you said and I believe you were correct. The cookies weren't the problem.

h ttp://rs289.rapidshare.com/files/70947297/Wise_Intelligent_-_Blessed_Be_The_Poor__2007_.rar
That is the site that is still giving me problems. I used a download manager to grab the link without actually attempting to download the file. It gives this as the download URL h ttp://rs289tl3.rapidshare.com/files/70947297/1522915/Wise_Intelligent_-_Blessed_Be_The_Poor__2007_.rar?mirror=on&x=63&y=50
When I tried to download the file with that URL I still got the same 404 error. I did more research and found that the end part of the url is a query string and I believe I'm supposed to send something back to the server through a "POST" to tell it I want the actual file location. I'm not sure though. Can someone clarify that for me. If I do have to write back to the server, what specifically am I supposed to write to the server and how do I get the actual file location back from the server. Any help will be appreciated.

[fishtoprecords]

I should have said before the errors occur. I was using two connections.

Don't do that.

Open one to get the request, and set the cookie in the response.

This is what servlets are designed to do. You should look carefully at them, and use them if you can.

[J-Live]

OK, I'll look into servlets for future projects. Do you know anything about the url string query question I was asking? That's the basically the last thing left for my current project.

[fishtoprecords]

You are supposed to parse the response from the server you first talk to, and follow any redirects it gives you.

It is extremely unreliable to manually follow links. Mirrors change. This is another "don't do that" case.

[J-Live]

Do you mean parse it like this example?

import java.net.*;
import java.io.*;

public class ParseURL {
public static void main(String[] args) throws Exception {
URL aURL = new URL("h ttp://java.sun.com:80/docs/books/tutorial"
+ "/index.html?name=networking#DOWNLOADING");
System.out.println("protocol = " + aURL.getProtocol());
System.out.println("authority = " + aURL.getAuthority());
System.out.println("host = " + aURL.getHost());
System.out.println("port = " + aURL.getPort());
System.out.println("path = " + aURL.getPath());
System.out.println("query = " + aURL.getQuery());
System.out.println("filename = " + aURL.getFile());
System.out.println("ref = " + aURL.getRef());
}
}

[fishtoprecords]

You seem to be focusing on the URL. That's not at all what I'm talking about.

You use the HTTP protocol to do a GET or POST to a host, and the URL describes it. The URL may contain query parameters.

You need to interact with the server, and get the response, and parse the response. You can't know what the response will be until you get it.

While the HttpUrlConnection hides a lot of the details, in this case, you need the details of the live interaction.

[J-Live]

How do I know what to send in a POST or what to ask for in a GET request? That's where I get lost. I have seen a few examples like this one but I don't know what I'm supposed to send. All I get from this example is that he's writing his name and e-mail to the server and getting the response from it.

URL url;
URLConnection urlConn;
DataOutputStream printout;
DataInputStream input;
// URL of CGI-Bin script.
url = new URL (getCodeBase().toString() + "env.tcgi");
// URL connection channel.
urlConn = url.openConnection();
// Let the run-time system (RTS) know that we want input.
urlConn.setDoInput (true);
// Let the RTS know that we want to do output.
urlConn.setDoOutput (true);
// No caching, we want the real thing.
urlConn.setUseCaches (false);
// Specify the content type.
urlConn.setRequestProperty
("Content-Type", "application/x-www-form-urlencoded");
// Send POST output.
printout = new DataOutputStream (urlConn.getOutputStream ());
String content =
"name=" + URLEncoder.encode ("Buford Early") +
"&email=" + URLEncoder.encode ("buford@known-space.com");
printout.writeBytes (content);
printout.flush ();
printout.close ();
// Get response data.
input = new DataInputStream (urlConn.getInputStream ());
String str;
while (null != ((str = input.readLine())))
{
System.out.println (str);
textArea.appendText (str + "\n");
}
input.close ();

[Norm]

what to send in a POST or what to ask for in a GET request

A POST is often done from a <FORM tag. The contents are as set by the tags in the <FORM.
A GET can also be done from a <FORM or it can just be a simple request for a file, ie no query string.

The contents of either of them depends on the HTML page that the server sent you that you are responding to. You can't make it up, it must be from the tags in the HMTL page. Your app will have to do some of the work that a browser does to parse the HTML and retrieve the strings you need to send a correct request to the server in response to the HTML page it sent you.

[Nicholas Jordan]

How do I know what to send in a POST or what to ask for in a GET request? That's where I get lost. I have seen a few examples like this one but I don't know what I'm supposed to send. All I get from this example is that he's writing his name and e-mail to the server and getting the response from it.

"&email=" + URLEncoder.encode ("*****@****-****.pit");

Screen emails in posts for one thing.

A GET request has information written in the address bar of the browser. For example:

http://nail.hammer.info/toolsteel/claw?logon=userizatwit

would be a place for people know nothing of getting hammered and wish to start their own e-business online by looking at a web page instead of swinging a hammer.

This paticular url, fabricated to be obviously bogus to all except people who are clueless when reading an address in the browser address bar, goes first to nail.hammer.info by DNS and sends to that server a request which contains toolsteel/claw which should be interpreted as a common filename and/or directory like normal computer use.

All after the question mark is to be what is called name value pairs.

String first = "abra";//
String second = "cadabra";//
String nvSeparator = "=";//
StringBuffer demonstration = new StringBuffer("nail.hammer.info/toolsteel/claw");
demonstration.append("?");//
demonstration.append("first");//
demonstration.append(nvSeparator);//
demonstration.append("=");//
demonstration.append("second");//
URL firstTryURL = new URL(demonstration.toString());//

Which now becomes visible in the address bar.

A POST has a pair of cr/lf's which would appear before request data - in essence what the client and server are doing is sending strings back and forth, with an end-of-line value ( 0x0a && 0x0d ) used to provide some of the information that the server and browser use for decisions on what to do with what comes and goes.

In a post request, the contents of a form:

<form action="nail.hammer.info"><input name="logon" value="enter hammerdown orginizations's famous citation here"><input type="submit"></form>

would then send "userizatwit" if that is what someone who does not know how to use a hammer knew what to type in to the input.

When the mouse moves over the "Submit" button, submission is just a click away.

Your sample code appears to have dot cgi in the code, and looks to me to be a sample from one of the Unice.

[fishtoprecords]

How do I know what to send in a POST or what to ask for in a GET request? That's where I get lost.

Then your early claim that you are nearly finished is completely wrong. You don't understand what you are doing. You can't claim to be nearly finished.

You can know the initial URL. Just as a browser knows it when you enter it in the address bar field. From this initial URL, you make your connection. But you have to read the results and process them. You must look at the status code.

While the official list is at the IETF, this has the basics:
List of HTTP status codes - Wikipedia, the free encyclopedia

[J-Live]

A POST is often done from a <FORM tag. The contents are as set by the tags in the <FORM.
A GET can also be done from a <FORM or it can just be a simple request for a file, ie no query string.

The contents of either of them depends on the HTML page that the server sent you that you are responding to. You can't make it up, it must be from the tags in the HMTL page. Your app will have to do some of the work that a browser does to parse the HTML and retrieve the strings you need to send a correct request to the server in response to the HTML page it sent you.

Thank you. I kind of discovered that myself last night. I modified the example I found and it gets the file I want now. I'm confused on how can I use setRequestProperty("Range", "bytes="...) on the response from the sever. Is it possible to use that method in a response to a POST(on the data that is being returned by the server)?

URL url;
HttpURLConnection urlConn;
DataOutputStream  printout;
DataInputStream   input;
 InputStream in;  
 RandomAccessFile out;
    public void donkey(){
        try {
            url = new URL("ht-tp://rs289cg.rapidshare.com/files/70947297/1550378/Wise_Intelligent_-_Blessed_Be_The_Poor__2007_.rar?mirror=on&x=45&y=44");
        } catch (MalformedURLException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
        try {
            // URL connection channel.
            urlConn = (HttpURLConnection) url.openConnection();
        } catch (IOException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
    // Let the run-time system (RTS) know that we want input.
    urlConn.setDoInput (true);
    // Let the RTS know that we want to do output.
    urlConn.setDoOutput (true);
    // No caching, we want the real thing.
    urlConn.setUseCaches (false);
      
        //urlConn.setRequestProperty("Range", "bytes=" + "-" + "4500");
      
        try {
            urlConn.setRequestMethod("POST");
        } catch (ProtocolException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
    // Specify the content type.
    
    urlConn.setRequestProperty("Connection", "keep-alive");    
    urlConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
        try {
            urlConn.connect();
        } catch (IOException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
        try {
            // Send POST output.
            printout = new DataOutputStream(urlConn.getOutputStream());
        } catch (IOException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
    String content = null;
        try {
        content = "name=dlf" + URLEncoder.encode(url.getQuery(), "UTF-8");
        } catch (UnsupportedEncodingException exp) {
        Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, exp);       }
    try {

            printout.writeBytes(content);
            printout.flush();
            printout.close();
        } catch (IOException exp) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, exp);
        }
    
        try {
            
            
            // Get response data.
            in = urlConn.getInputStream();
            
        } catch (IOException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
    String str;
        try {
            out = new RandomAccessFile("something.rar", "rw");
        } catch (FileNotFoundException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
    byte[] buffer = new byte[1024];
    int downloaded = 0;
        try {
            int numRead;
		
             
                   while ((numRead = in.read(buffer)) != -1) {
               
				out.write(buffer, 0, numRead);
				downloaded += numRead;
                  
                        
                  System.out.println("Downloaded " + downloaded);
            }
        } catch (IOException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }
        try {
            in.close();
            out.close();
        } catch (IOException ex) {
            Logger.getLogger(fuckifiknow.class.getName()).log(Level.SEVERE, null, ex);
        }

[fishtoprecords]

I'm confused on how can I use setRequestProperty("Range", "bytes="...) on the response from the sever. Is it possible to use that method in a response to a POST(on the data that is being returned by the server)?

Why be confused? What does the javadoc say on that function?

In general, you have to be ready for whatever the server sends. True for both POST and GET. The server may, or may not, pay attention to what you ask for.

Its a lot easier to just accept whatever you get, and process it. Or stream it to a temp file if its too huge.