java.io.IOException: Premature EOF

Hi guys!

I am getting a java.io.IOException: Premature EOF error in some urls (usually, the ones with a huge content), after executing the following peace of code. Any ideas?

Code:

---

public static void urlReader(String url, String path, String fileName) {
        String inputLine = null;
       
        URL x = null;
        BufferedReader in = null;
       
        createFile(path, fileName);

        try {
            // Create file
            FileWriter fwstream = new FileWriter(path + fileName);
            BufferedWriter out = new BufferedWriter(fwstream);

            x = new URL(url);
            in = new BufferedReader(new InputStreamReader(x.openStream()));

            while ((inputLine = in.readLine()) != null) {

                if (inputLine.length() > 0) {
                    //System.out.println(inputLine + "\n");
                    out.write(inputLine + "\n");
                }
            }

            // Close the input stream
            in.close();

            // Close the output stream
            out.close();
        } catch (IOException ex) {
            System.out.println(ex);
            System.out.println("Application aborted while function \"urlReader();\" was executing!");
            System.out.println("url = \"" + url + "\"\n");
            System.out.println("inputLine = \"" + inputLine + "\"\n");
            //System.exit(-1);
        }
    }

---

I think the problem is you are trying to read the content in a line oriented way, that is, if the content being read is a text file and the file ends with the end of line character, this will work, but if the file is a binary file, or does not end with the end of line character, the while() loop will be waiting to see that end of line, but the stream has already ended, and the sender might try to close the connection, and that will cause this premature EOF exception.

So instead of using the line approach, use the read into a buffer approach,

Code:

---

static int BUFFER_SIZE=1024;
byte[] buffer = new byte[BUFFER_SIZE]; // or some other size,
int charsRead = 0;
while ( (charsRead  = in.read(buffer,0,BUFFER_SIZE) != -1) {
  out.write(buffer,0, charsRead);
}

---

where here we don't have to re-add the "\n" like before when reading lines.

note the read(buffer, offset, max length) will return the actual bytes read, which we test if this is -1 this means the end of stream has been reached. And then we use this bytes read amount to tell the write(buffer, offset ,max length) to only try to write this many bytes from the buffer. This allows for less then the buffer size amount of bytes to be read at a time, such as the remainder of things in the file that are less than the buffer size as we get near the end of the file. For more details, see the Java API documentation on InputStream and OutputStream.

Hi mr. travishein and thanks for your reply. Unfortunately, the same error occurs again and again for different values of BUFFER_SIZE variable. Please, take a look on the following peace of code and tell me if there is any wrong with it.

Kind regards,
CrazyVag.

Code:

---

static int BUFFER_SIZE = 1024;
//static byte[] buffer = new byte[BUFFER_SIZE]; // or some other size.
static char[] buffer = new char[BUFFER_SIZE]; // or some other size.
static int charsRead = 0;

public static void urlReader(String url, String path, String fileName) {
        String inputLine = null;
       
        URL capital = null;
        BufferedReader in = null;
       
        createFile(path, fileName);

        try {
            // Create file
            FileWriter fwstream = new FileWriter(path + fileName);
            BufferedWriter out = new BufferedWriter(fwstream);

            capital = new URL(url);
            in = new BufferedReader(new InputStreamReader(capital.openStream()));

//            while ((inputLine = in.readLine()) != null) {
//                // Removes leading and trailing white space from a string.
//                inputLine = inputLine.trim();
//
//                if (inputLine.length() > 0) {
//                    //System.out.println(inputLine + "\n");
//                    out.write(inputLine + "\n");
//                }
//            }

            while ( (charsRead = in.read(buffer, 0, BUFFER_SIZE)) != -1 ) {
                // Removes leading and trailing white space from a string.
                //inputLine = inputLine.trim();

                //if (inputLine.length() > 0) {
                    //System.out.println(inputLine + "\n");
                    //out.write(inputLine + "\n");
                //}
                out.write(buffer, 0, charsRead);
                //System.out.println(inputLine + "\n");
            }

            // Close the input stream
            in.close();

            // Close the output stream
            out.close();

        } catch (IOException ex) {
            System.out.println(ex);
            System.out.println("Application aborted while function \"urlReader();\" was executing!");
            System.out.println("url = \"" + url + "\"\n");
            System.out.println("inputLine = \"" + inputLine + "\"\n");
            //System.exit(-1);
        }
    }

---

Okay. Here we are!

I added the following command (i.e. ex.printStackTrace();) inside the catch block in order to show you the stack trace.
I really concern about the behavior of my application because the "stuck" comes from different urls in each execution...
Also, it is worth to notice that, after the installation of netBeans and the app-execution in a new machine, the app completed successfully!
But, this doesn't happen again! What the !@#$ is going on? How can I solve the bug?
Some guys told me about the following improvement but neither this worked out!

Code:

---

while (!in.ready())
{
Thread.sleap(100); // wait for stream to be ready.
}

---

Code:

---

run:
Application aborted while function "urlReader();" was executing!
java.io.IOException: Premature EOF
java.io.IOException: Premature EOF
        at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538)
        at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582)
        at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669)
        at java.io.FilterInputStream.read(FilterInputStream.java:116)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2512)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
        at java.io.InputStreamReader.read(InputStreamReader.java:167)
        at java.io.BufferedReader.fill(BufferedReader.java:136)
        at java.io.BufferedReader.readLine(BufferedReader.java:299)
        at java.io.BufferedReader.readLine(BufferedReader.java:362)
        at x.application.urlReader(application.java:429)
        at x.application.downloadPages(application.java:410)
        at x.application.main(application.java:46)
Java Result: -1
BUILD SUCCESSFUL (total time: 2 minutes 10 seconds)

---

Hmm, I wonder if it is something to do with the remote site, or if the location where you are connecting from is going through a HTTP proxy server, where there might be some kind of administrative limit on the amount of time that is allowed for a given open connection. In that, these URLs that fail, do they consistently fail, or do they ever sometimes work ?

A couple of points from this code as you last posted it
- how come the buffer and bytes read are static, and outside of the method. The buffer should be within the scope of the method, that is to make it thread safe. more than one concurrent invocation of this function as it is would break, as they would compete to read and write from this single static buffer in a chatotic manner.
- i'm not entirely sure why you are reading char's , instead of bytes, is it that you want to read a text-based resource and remove the extra whitespace. - is it safe to assume these URLs' you would be fetching would always be text base anyway ?
- Also, if the url's are from a HTTP server, using the raw read approach like this, wouldn't this cause the HTTP headers (if any) from the remote HTTP server to be stuffed into file contents that we are writing out. in that, would we not need to have something to parse, or at least skip over the http headers for us ?

I'm not very experienced with using the standard Java URL to input stream things, I'm sure there is someone who might have a simple "just do this" setting to make this work better, such as turning on some god awful internal Java VM system property that will magically cause the Sun URL HTTP IO to somehow start working. But my experience with this Java stnadard URL HTTP fetching stuff has always been very bad, and only breaks for me if we try to do something only slightly different.

For HTTP fetching, I usually just use that Apache commons-httpclient project, which means adding commons-httpclient.jar (and their dependencies commons-codec, commons-logging if not already there) to the class path, such as WEB-INF/lib in a web application module, or your classpath for command line operations. Assuming that adding this additional API is OK,

But in the interest of trying a self contained example first, How about if we try this Java URL API way one more time, and cast the URL into a HttpURLConnection, in case it has some better handling for the HTTP headers, handling some time outs with connections, and such.

Code:

---

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;

import org.junit.Test;

/**
 *Created on Jan 25, 2010
 */

/**
 * @author thein
 *
 */
public class ReadURLCharactersToFile {


  static int BUFFER_SIZE = 1024;

  /**
  * Reads a URL to a local file
  * @param url the url to be read
  * @param path the folder component of the file to be written
  * @param fileName the file name, following the file path
  * @return the total number of characters read.
  * @throws IOException if an error occurs
  */
  public static int urlReader(String url, String path, String fileName) throws IOException {

    Reader in = null;
    Writer out = null;
    int totalCharsRead = 0;
    try {
      URLConnection urlConnection = new URL(url).openConnection();
      if (urlConnection instanceof HttpURLConnection) {
       
        HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
   
        in = new BufferedReader(new InputStreamReader(httpUrlConnection.getInputStream()));
        out = new BufferedWriter(new FileWriter(new File(path,fileName)));
     
        char[] buffer = new char[BUFFER_SIZE];
        int charsRead = 0;
   
        while ( (charsRead = in.read(buffer, 0, BUFFER_SIZE)) != -1 ) {
          out.write(buffer, 0, charsRead);
          totalCharsRead += charsRead;
        }
      }
      else {
        System.err.println("unhandled URL connection type, we currently only support HttpURLConnection. URL=" + url);
      }
    }
    finally {
      if (in != null) {
        try {
          in.close();
        }
        catch (IOException ex) { /* empty */ }
      }
      if (out != null) {
        try {
          out.close();
        }
        catch (IOException ex) { /* empty */ }
      }
    }
    return totalCharsRead;
  }

  // assumes you have Junit 1.4+ jar file
  @Test
  public void testReadURL() {
    try {
      String theUrl ="http://www.google.com";
      int chars = ReadURLCharactersToFile.urlReader(theUrl, "/tmp", "page1.html");
      System.out.println("read " + chars + " from URL " +  theUrl);
    }
    catch (IOException ex) {
      System.err.println(ex.getMessage());
      ex.printStackTrace();
    }
  }
}

---

Where the second method is my Junit run it method that just invokes the first modified urlReader() method.

No.
Neither this does work...
As a newbie, I am not familiar with the stack trace.
Is there any way of using this kind information?