Http Buffer issue... chunked?

[jlew]

I am writing this little java code (not an applet, but used as its own jar file) that talks to a web server using http but at times gets weird responses with VERY LONG LINES without line returns.

Code:

Socket socket    = new Socket( websitehost, 80);
OutputStream os   = socket.getOutputStream();
PrintWriter out   = new PrintWriter( os, true );
BufferedReader in = new BufferedReader( new InputStreamReader( socket.getInputStream() ) );

//Talk to Server
out.println("GET " + URL_PREFIX + "&data=" + request + " HTTP/1.1");
out.println("Host: " + websitehost);
out.println("User-Agent: " + agent);
out.println("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
out.println("Accept-Language: en-us,en;q=0.5");
out.println("Accept-Encoding: deflate");
out.println("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7");
out.println("Keep-Alive: 300");
out.println("Connection: keep-alive");
out.println("Referer: " + referer);
out.println("Cookie: pass=" + PASSW + "; email=" + EMAIL);
out.println();

 // read the response
boolean loop = true;
StringBuffer sb = new StringBuffer(8096);

while (loop) {
  if ( in.ready() ) {
     int i=0;
     while (i!=-1) {
        i = in.read();
        sb.append((char) i);
     }
     loop = false;
  }
  
  Thread.currentThread().sleep(50); //Don't remember why that is there
  }

   socket.close();
   return sb.toString();

The problem is this website from time to time gives out HUGE lines of code with no breaks, and for some reason (don't know if it is the string buffer or something) when the line gets so many characters it adds in a line break, a few digits and anther line break and then it is reading again as if nothing happened

HTML Code:

HTTP/1.1 200 OK
Date: Sun, 21 Oct 2007 18:08:53 GMT
Server: Apache/1.3.33 (Debian GNU/Linux) mod_throttle/3.1.2 mod_gzip/1.3.26.1a mod_fastcgi/2.4.2
Pragma: no-cache
Expires: Tue, 26-Oct-2000 12:00:00
Vary: *
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=ISO-8859-1

...

<img sr
2000
c=http://

And with that img sr break there are more then this, but im just showing you this because you don't need to see the entire output to get the idea.

Anyone have any ideas? Im thinking maybe its in the encoding returned "chunked" im not sure any ideas or solutions would be great.

I need to be able to send my own headers that is why i am using this method... i have to emulate browsers and junk's headers in this script as well as get the data.

[jlew]

The problem has to do with chunked data... Does anyone know a simple fix on how to tell it to read the chunked data and ignoring the chunked size flags. I was looking up this issue and what that \n2000\n is some sort of flag or something that when converted to hex it tells the next chunk size. Like in my code right now i see 2000 then the begin of html, after x number of bytes or chars or something it stops sends me a new flag saying how big the next chunk will be and then continues... Does anyone know how to handle chunked input?

I can't seem to find a really good strait forward answer on how to do this without having to include a bunch of server library which i don't have.