OutOfMemory error occurs, but profiler shows half the heap is free?!?!

[v.svetoslavov]

Hello!

So here is my situation: I have a web application (MyFaces+Hibernate) running on BEA WebLogic 8.1 (either with Sun JVM 1.4.2 or the included JRockit VM). While profiling the application, trying to find memory leaks, I came to an OutOfMemory error, while the profiler showed that half the heap was free (the other half could not be freed).

So my question is this: Are there any common reasons for situations like this? The only things that come to my mind are:
a) Some very big method needed more than the available heap memory
b) The profiler is wrong
c) I miss something

Any help would be appreciated!
Thanks in advance.

Kind Regards,
V. Svetoslavov

[Steve11235]

It's possible that the stack ran out of memory, rather than the heap. That's just a guess...

Another guess is that the heap became fragmented to a point where it could not be defragmented. I *think* the JVM tries to defragment the heap.

I think your initial inclination is correct; something is holding unnecessary references or instantiating way too many objects...

[toadaly]

You will get an OutOfMemory error when you attempt to allocate more than is available. If half the heap is shown available, that most likely means something is trying to allocate something really big, rather than a slow memory leak. Add a stack trace if you can, to figure out why.

[serjant]

Either increase your JVm memory , or improve your multi-threaded program. Try to know out in which thread this occurs, and try to synchronize the threads.

[v.svetoslavov]

Either increase your JVm memory , or improve your multi-threaded program. Try to know out in which thread this occurs, and try to synchronize the threads.

This is a web application and it is WebLogic that handles the threads, the webapp itself does not use threads (at least not in the code I wrote, I cannot guarantee for MyFaces and/or Hibernate and the other frameworks used)

[v.svetoslavov]

It's possible that the stack ran out of memory, rather than the heap. That's just a guess...

What exactly does this mean? Too many methods call each other? What is to do in such situation (which is not very probable I think, because such thing should make the code break every time it is called, while my situation is different.

Another guess is that the heap became fragmented to a point where it could not be defragmented. I *think* the JVM tries to defragment the heap.

Again, what can I do to improve such a problem?

[Steve11235]

OK. I wrote a simple program that tries to run out of memory.

package test;

import java.util.ArrayList;
import com.coultech.utility.ExceptionUtil;

public class MemoryAbuser {
	public static void main(final String[] args) {
		MemoryAbuser instance = new MemoryAbuser();
		instance.run();
	}

	private ArrayList<Long> _LongList = new ArrayList<Long>(100000);

	public void run() {
		long l = 0;
		Long aLong = null;
		try {
			for (l = 0; l < 10000000; l++) {
				aLong = new Long(l);
				_LongList.add(aLong);
			}
		}
		catch (Throwable t) {
			ExceptionUtil.printStackTrace(t);
			System.out.println("Entries: "
					+ l
					+ " Total: " +
					+ Runtime.getRuntime().totalMemory()
					+ " Free: "
					+ Runtime.getRuntime().freeMemory());
		}
	}
}

Here's the output:

2009-01-30_00.10.27.078: 
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Unknown Source)
	at java.util.Arrays.copyOf(Unknown Source)
	at java.util.ArrayList.ensureCapacity(Unknown Source)
	at java.util.ArrayList.add(Unknown Source)
	at test.MemoryAbuser.run(MemoryAbuser.java:22)
	at test.MemoryAbuser.main(MemoryAbuser.java:13)
Entries: 2,562,932 Total: 66,650,112 Free: 15,194,136

Obviously, there is plenty of heap space left, yet is gives a heap error.

All I an say is that out of memory *seems* to happen before very last scrap of the heap is used...

[toadaly]

That's an interesting example Steve. I'm kind of surprised by it. You have to wonder why the VM is reserving that 15M and not letting you use it.