Memory in java doesn't correspond with eclipse.ini

[christophe]

Hello everyone,

I have to make at least 2 really big objects (1 int[13800][13800] and 1 int[13800][9980]).

Therefore I need a lot of memory (more then 1GB).

I tried to change /etc/eclipse.ini (I use Ubuntu 10.04) and it looks now like this:

-startup
plugins/org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.0.200.v20090519
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
3072m
-vmargs
-Xms2048m
-Xmx3072m
-Dorg.eclipse.equinox.p2.reconciler.dropins.directory=/usr/share/eclipse/dropins

I have 4GB of ram in this and swapping should also be used if needed.
If I check the following:

public static void main(String[] args) {
		System.out.println(Runtime.getRuntime().maxMemory());
		System.out.println(Runtime.getRuntime().freeMemory());
		System.out.println(Runtime.getRuntime().totalMemory());
}

I get following return

920911872
61738824
62062592

These are numbers expressed in bytes, so it is a lot less as I try to get.

I hope someone can give me a clue.

Thanks in advance,
Christophe

[berkeleybross]

Have you tried just running eclipse with:

eclipse -vmargs -Xms2048m -Xmx3072m

You may also want to check out Eclipse and memory settings » Max's Blog

(its not my blog, its just a random one i found which helped sort out my problem a while back :P)

Berkeleybross

[christophe]

Thanks berkeleybross for looking into my problem.
I also used this file to get this far, but it doesn't work for me (even command line).

Even if I try with less memory maxMemory() etc. keeps the same

eclipse -vmargs -Xms128m -Xmx256m

[Eranga]

Therefore I need a lot of memory (more then 1GB).

Do you really believe that it's a good idea, or a good design?

[berkeleybross]

--launcher.XXMaxPermSize
3072m

I dont know much about permsize but what i understand is its seperate from heap space and i think its assigned before heap space.

The website i linked to shows a permsize of about 128m. Have you tried setting it at about that? I've got a feeling its ninja'ing your memory leaving nothing for the heapspace.

[christophe]

@Eranga:
I need to parse 9980 documents and extract all person names out of these. With this I have to make a term-document matrix in which every cell contains the term-frequency in a specific document. As there are 13840 unique names this is a 13840x9980 matrix.
In a next step I have to calculate the term-term correlation matrix, so this becomes a 13840x13840 matrix.
This will be used to try some clustering algorithms.

As you can see this project for school isn't really about 'good' design but more about good algorithms. If you can give me a good alternative to calculate these matrices I am eager to hear

[Eranga]

I'm not exactly clear what you mean by "parse 9980 documents". How did you pass all those things? And also once you process documents how did you start to build the matrix? Did you write them into a file or keep in heap? I think it is.

[christophe]

setting XXMAxPermSize to 128 or 256 doesn't change anything either.

[christophe]

@Eranga
I save and load them from a data file after creating it (serialized objects) so I only have the objects I need loaded.
After loading the term-document matrix totalMemory() tells me it uses 594608128 bytes so this works. The problem is that i need to create another big matrix while having the other object still in memory.

[berkeleybross]

I really cant think of much to help you...

Really stabbing in the dark, but have you tried running just the java file via command line instead of through eclipse?
May see if it really is a problem with the ini file.

Also have you tried looking at the task manager to see if linux thinks the memory is a different size than java?

And can you tell us the exact error message that occurs? Out of memory errors dont usually tell much but every little helps..

[Tolls]

Just a question, but is this a 64 bit system with a 64 bit Java?
If not then you actually only have about 1.5G max, some of which is used by the JVM itself and so isn't heap.

Also, as has been pointed out, you don't want perm size to be anywhere near that amount. Perm space is for storing class information.

One more thing, you really don't want to be using swapping with Java. It really, really slows down the garbage collection.

Oh yes, and when running your code is it actually running inside the eclipse JVM or another one? Because I think you should be setting this in the individual run configuration for the project and not for the Eclipse launcher...

[Eranga]

@Eranga
I save and load them from a data file after creating it (serialized objects) so I only have the objects I need loaded.
After loading the term-document matrix totalMemory() tells me it uses 594608128 bytes so this works. The problem is that i need to create another big matrix while having the other object still in memory.

How about keeping them in a database if you really want to use such a huge memory space.