Results 1 to 6 of 6
Like Tree1Likes
  • 1 Post By Tolls

Thread: Deserialize Large Object from Url: StreamCorruptedException

  1. #1
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    4,041
    Rep Power
    10

    Question Deserialize Large Object from Url: StreamCorruptedException

    I have an application with a bunch of optional modules. These modules can be quite large, and a user can pick and choose which modules to run. I also want to be able to add modules without making users download the application again. So I'm trying to store the modules as serialized objects on the web and create an online properties file that lists these possible modules along with the url of the serialized object. All the program does at the beginning is read the properties file and show the user the optional modules. Then when the user chooses which modules to run, the application downloads the serialized objects from the url, deserializes them, and runs them.

    This idea mostly works. However, I want to add loading bars as the modules are downloaded. To do that, I'm manually reading the bytes from the serialized object, then feeding those bytes to an ObjectInputStream.

    That also works, as long as the serialized objects are relatively small (the problem occurs at 64,000,109 bytes but not with smaller files, and my actual modules can be much larger). But once I get over some to-be-determined threshold, I get the following Exception:

    Java Code:
    Exception in thread "main" java.io.StreamCorruptedException: invalid type code: 00
    	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1374)
    	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
    	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
    	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
    	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
    	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
    	at SerializeTest.readAndDisplayProgress(SerializeTest.java:58)
    	at SerializeTest.main(SerializeTest.java:81)

    I've thrown together an SSCCE that demonstrates exactly what I'm doing. I create the serialized Object file using the createAndSerialize() function, then I upload it to my S3 site. Both my file system and the S3 url contain the same exact file. I then comment out one of the read() function calls to test the different options. They all work, except when I try to read a large file from a url.

    The thing is, it gets through the actual downloading just fine. It's only when it goes to deserialize the object that it throws the Exception. Hopefully I'm doing something dumb, but I can't figure this out.

    Java Code:
    import java.io.ByteArrayInputStream;
    import java.io.ByteArrayOutputStream;
    import java.io.FileInputStream;
    import java.io.FileOutputStream;
    import java.io.InputStream;
    import java.io.ObjectInputStream;
    import java.io.ObjectOutputStream;
    import java.io.Serializable;
    import java.net.URL;
    
    public class SerializeTest implements Serializable{
    
    	private static final long serialVersionUID = 1L;
    
    	String value;
            //this is just to make the file large, change it to change the file size
    	double[] d = new double[8000000];
    
    	public SerializeTest(){}
    
    	public SerializeTest(String value){
    		this.value = value;
    	}
    
    	public static void createAndSerialize() throws Exception{
    
    		SerializeTest test = new SerializeTest("testing 123");
    
    		FileOutputStream fileOut =  new FileOutputStream("C:\\Users\\kworkman\\Desktop\\test.ser");
    		ObjectOutputStream out = new ObjectOutputStream(fileOut);
    		out.writeObject(test);
    		out.flush();
    		out.close();
    	}
    
    	public static void readAndDisplayProgress(InputStream input) throws Exception{
    
    		//size of input file is known ahead of time
    		int size = 64000109;
    		ByteArrayOutputStream bos = new ByteArrayOutputStream();
    
    		int totalRead = 0;
    		while(totalRead < size){
    			System.out.println("Progress: " + (int)(100*(totalRead/(double)size)) +"%");
    			//actual buffer is larger, but this shows the progress bar idea
    			byte[] buffer = new byte[16384];
    			int read = input.read(buffer);
    			bos.write(buffer);
    
    			totalRead += read;
    		}
    
    		bos.flush();
    		ByteArrayInputStream byteInput = new ByteArrayInputStream(bos.toByteArray());
    
    		ObjectInputStream in = new ObjectInputStream(byteInput);
    
    		//this is the line that throws the StreamCorruptedException
    		SerializeTest test = (SerializeTest) in.readObject();
    		System.out.println("Value: " + test.value);
    	}
    
    	public static void readWithoutDisplayingProgress(InputStream input) throws Exception{
    		ObjectInputStream in = new ObjectInputStream(input);
    		SerializeTest test = (SerializeTest) in.readObject();
    		System.out.println("Value: " + test.value);
    	}
    
    	public static void main(String... args) throws Exception{
    
    		//createAndSerialize();
    
    		//this works great!
    		//FileInputStream fileIn = new FileInputStream("C:\\Users\\kworkman\\Desktop\\test.ser");
    		//readAndDisplayProgress(fileIn);
    
    		
    		InputStream urlIn = new URL("http://s3.staticvoidgames.com/LudumDareDataMining/test.ser").openStream();
    
    		//this also works, but doesn't give me progress
    		//readWithoutDisplayingProgress(urlIn);
    
    		//this generates a StreamCorruptedException
    		readAndDisplayProgress(urlIn);
    	}
    
    }
    I'm also very open to suggestion on different approaches to my main goal of showing progress while downloading serialized objects off the web. This is what I came up with, but there might be (probably is) a better approach.

    As always, thanks for your time!
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  2. #2
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,224
    Rep Power
    20

    Default Re: Deserialize Large Object from Url: StreamCorruptedException

    Java Code:
    bos.write(buffer);
    It looks like you are writing out the whole buffer.
    You have to take into account the amount actually read, as there is no guarantee that the buffer will actually be filled.
    KevinWorkman likes this.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  3. #3
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    4,041
    Rep Power
    10

    Default Re: Deserialize Large Object from Url: StreamCorruptedException

    Quote Originally Posted by Tolls View Post
    Java Code:
    bos.write(buffer);
    It looks like you are writing out the whole buffer.
    You have to take into account the amount actually read, as there is no guarantee that the buffer will actually be filled.
    Thanks for the reply. I had to think about that one for a minute, but you mean this?

    Java Code:
    int totalRead = 0;
    		while(totalRead < size){
    			System.out.println("Progress: " + (int)(100*(totalRead/(double)size)) +"%");
    			//actual buffer is larger, but this shows the progress bar idea
    			byte[] buffer = new byte[16384];
    			int read = input.read(buffer);
    			bos.write(buffer, 0, read);
    
    			totalRead += read;
    		}
    Unfortunately I'm behind a firewall right now so I can't test this yet, but it seems to make sense.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  4. #4
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,224
    Rep Power
    20

    Default Re: Deserialize Large Object from Url: StreamCorruptedException

    That's the one.

    The usual loop is:
    Java Code:
    byte[] buffer = new byte[whatever size];
    int read;
    while ((read = input.read(buffer)) > -1) {
        output.write(buffer, 0, read);
    }
    No point creating a new buffer object each time round the loop.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  5. #5
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    4,041
    Rep Power
    10

    Default Re: Deserialize Large Object from Url: StreamCorruptedException

    Awesome, thanks. I can't wait to test this (I spent *way* too much time banging my head against this last night), but I won't be home for another 12 hours (working full time and going to school for my master's is fun).

    My loop did look more like that originally (minus the part that takes into account how much was read when writing), but I modified it to what you see above because I thought maybe there was an issue with the buffer holding old data (which turns out to be almost-sorta-but-not-really the actual problem).

    Seriously, thanks again.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  6. #6
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    4,041
    Rep Power
    10

    Default Re: Deserialize Large Object from Url: StreamCorruptedException

    Just wanted to update and say that this seems to be working perfectly now. Thanks again.
    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

Similar Threads

  1. StreamCorruptedException what?
    By Aubanis in forum New To Java
    Replies: 4
    Last Post: 04-16-2011, 07:08 AM
  2. StreamCorruptedException
    By Fortu in forum Advanced Java
    Replies: 1
    Last Post: 04-10-2011, 10:56 PM
  3. StreamCorruptedException
    By wikisb in forum New To Java
    Replies: 3
    Last Post: 10-28-2010, 08:16 AM
  4. StreamCorruptedException
    By cristo_haris in forum Advanced Java
    Replies: 11
    Last Post: 04-20-2009, 04:44 PM
  5. how to deserialize multiple objects in a file
    By xcallmejudasx in forum Advanced Java
    Replies: 11
    Last Post: 12-16-2008, 06:29 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •