Results 1 to 10 of 10
  1. #1
    kslnet is offline Member
    Join Date
    May 2010
    Location
    Cambridge, MA
    Posts
    5
    Rep Power
    0

    Default Java serialization - why do I run out of memory upon restore?

    Any serialization experts able to help me on this?

    I have a large array of MyClass objects with the following structure:

    Java Code:
    private class MyClass implements Serializable
    {
        int a;
        int b;
        // lots of other primitive type fields
        transient HashSet<SecretClass> classSet = null;
    
        private MyClass (int a, int b) {
            this.a = a;
            this.b = b;
            this.classSet = new HashSet<SecretClass>();
    }
    There's no memory problem creating the array. I want to serialize it to create a cache so I don't have to repeat the computation in future runs. Now, SecretClass isn't serializable, so I instead I store an integer "somefield" from which I can later restore it:

    Java Code:
    for (int i = 0; i < arraySize; i++) {
        MyClass temp = MyClassArray.get(i);
        cacheFile.writeObject(temp);
        cacheFile.writeObject(temp.classSet.size());
        for (SecretClass sc : temp.classSet) { 
            cacheFile.writeObject(sc.someField);
        }
    }
    Now, when I restore it with

    Java Code:
    for (int i = 0; i < arraySize; i++) {
         restoredClass = (MyClass) cacheFile.readObject();
         restoredClass.classSet = new HashSet<SecretClass>();
         int hashSize = (Integer) cacheFile.readObject();
         for (j = 0; j < hashSize; j++) {
              int someField = (Integer) cacheFile.readObject();
              SecretClass sc = SecretClass.getObjFromId(someField);
              restoredClass.classSet.add(sc);
         }
    }
    This gives me a java.lang.OutOfMemoryError: Java heap space
    I clearly have enough memory to store the array because I can generate it, so why can't I restore it? I'm pretty sure this has to do with how I'm handling the HashSet<SecretClass>, but I can't figure it out.

    Thanks in advance.
    Karin

  2. #2
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,581
    Rep Power
    25

    Default

    No ideas yet.
    What is the value of hashSize returned by readObject()?
    How many times do you go thru the read loop before the error?

  3. #3
    kslnet is offline Member
    Join Date
    May 2010
    Location
    Cambridge, MA
    Posts
    5
    Rep Power
    0

    Default

    Thanks for replying.

    The hashSize is small, usually 0-3. I can read through about 700K objects out of 750K.

    I realize that a HashSet is probably overkill. I had similar problems with ArrayList. I could choose not to use any dynamic structures and just preallocate an array of 10 SecretClass objects, but I was trying to save memory (I don't know what the overhead of a HashSet is). And I'd really like to be able to use the cache I've already created. :o

  4. #4
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,581
    Rep Power
    25

    Default

    What are the values for arraySize? How far into that count do you get?
    Last edited by Norm; 05-20-2010 at 08:10 PM.

  5. #5
    kslnet is offline Member
    Join Date
    May 2010
    Location
    Cambridge, MA
    Posts
    5
    Rep Power
    0

    Default

    hashSize is 0-3, but arraySize is about 750K.

  6. #6
    iluxa is offline Senior Member
    Join Date
    Mar 2010
    Posts
    266
    Rep Power
    5

    Default

    1. HashSets are expensive. ArrayLists are less expensive. You may want to re-consider ArrayLists.
    2. Default size of a HashSet is 16, and your hashSize is 0..3. 13..16 elements of HashSet are wasted.


    perhaps try something like this for restore:

    Java Code:
    for (int i = 0; i < arraySize; i++) {
         restoredClass = (MyClass) cacheFile.readObject();
         int hashSize = (Integer) cacheFile.readObject();
         if(hashSize > 0) { // otherwise dont even create the object!
              // allocate only the memory you need
              restoredClass.classSet = new ArrayList<SecretClass>(hashSize); 
              for (j = 0; j < hashSize; j++) {
                   int someField = (Integer) cacheFile.readObject();
                   SecretClass sc = SecretClass.getObjFromId(someField);
                   restoredClass.classSet.add(sc);
              }
         }
    }

  7. #7
    kslnet is offline Member
    Join Date
    May 2010
    Location
    Cambridge, MA
    Posts
    5
    Rep Power
    0

    Smile

    iluxa, thank you so much...of course you're right, and that fixed the problem.

    I still don't see why I run out of memory when I can generate the data to begin with, but as we engineers tend to do, I'm moving on 'cause it works. :p

  8. #8
    j2me64's Avatar
    j2me64 is offline Senior Member
    Join Date
    Sep 2009
    Location
    Zurich, Switzerland
    Posts
    962
    Rep Power
    6

    Default

    Quote Originally Posted by kslnet View Post
    iluxa, thank you so much...of course you're right, and that fixed the problem.

    I still don't see why I run out of memory when I can generate the data to begin with, but as we engineers tend to do, I'm moving on 'cause it works. :p

    when you define your collection inside a class then you can save all objects inside it at once without a for loop, same when you deserialize it.

  9. #9
    kslnet is offline Member
    Join Date
    May 2010
    Location
    Cambridge, MA
    Posts
    5
    Rep Power
    0

    Default

    You're right, j2me64, and I'm aware of that, but it doesn't work when the classes in my collection are not serializable.

  10. #10
    iluxa is offline Senior Member
    Join Date
    Mar 2010
    Posts
    266
    Rep Power
    5

    Default

    Was thinking some more about it... Maybe Java de-serialization is dumb and is creating its own object of some kind for every object it's de-serializing, and holds on to these until the very end... cause otherwise your problem makes no sense - if you had it in memory once, you should be able to get it back into same amount of memory...

Similar Threads

  1. serialization in java
    By sunithamm in forum Advanced Java
    Replies: 5
    Last Post: 01-25-2010, 02:11 PM
  2. Java Serialization
    By Cbani in forum New To Java
    Replies: 18
    Last Post: 01-20-2010, 12:31 PM
  3. Serialization - writing/reading to memory
    By ajeeb in forum Advanced Java
    Replies: 3
    Last Post: 01-27-2009, 03:07 PM
  4. Replies: 0
    Last Post: 04-04-2008, 02:47 PM
  5. Replies: 7
    Last Post: 08-11-2007, 09:44 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •