Garbage Collector tuning

Hi,
I need some help to run my java application which processes millions of records. I do this in a batch fashion where I process around 10000 records in each batch.
I have to create multiple objects and release references to most of them during each batch.

Java Garbage collector(GC) doesn't seem to be actively freeing objects and hence I have my application getting very slow after 15-20 mins when memory is almost fully consumed.

I tried tuning GC with: -XX:ExplicitGCInvokesConcurrent
-XX:+UseAdapativeGCBoundary
but they just postponed the delay problem.

I am sure java should have some option to handle this issue.
Can somebody suggest any idea?

thanks,
Dilip

This doesn't seems to be Garbage Collection problem alone.

I sense some memory leak in your program. Review your code for any references not freed up. You can use any profiler to figure out the exact object which is not getting freed up. As you might be using threads to process each record, several threads eventually consume your whole memory.

Just out of curiosity, if you put System.GC() into your loop, does memory still fill up after 15-20 min?

Quote:

---

Originally Posted by lovelesh

This doesn't seems to be Garbage Collection problem alone.

I sense some memory leak in your program. Review your code for any references not freed up. You can use any profiler to figure out the exact object which is not getting freed up. As you might be using threads to process each record, several threads eventually consume your whole memory.

---

This.
If the memory is slowly being consumed like this then you have a leak.

First get a heap dump from your app as it's slowing down (so after 15 minutes or so) and then stick that in an analyser, like Eclipse MAT. This should highlight the objects that are the likely root cause of your problem. From there you should be able to figure out why they aren't being cleared out.

We can make guesses here, but until you've done that step that's all they'll be...guesses.

I checked with the mem dumps.
The leak report says no leak.

I also tried using system.gc very frequently, but it doesn't seem to help either.

I got an out of memory error after 20-30 mins and the dump size was 6.1 GB.
I am unable to load it in eclipse as MAT tool in eclipse says: Error: Java heap space.

From the GCDetails printed, I see that GC is blocking the application for around 9-10 seconds very often after 15-20 mins of starting the program.
My program is expected to hold references to some memory objects initially.
So initially I expect GC to free fewer objects and later on GC should be able to free huge chunk of short lived objects.

The number of objects I hold reference to is finite and much lesser than what my physical memory(8GB) can accommodate.

Any advice?

Well, as far as I know, java can still only use 2GB max per instance of a VM. You might try adding -Xmx2048m -Xms1024m or something similar to your launch arguments to see if it helps (the defaults are much much lower). However, if you are only running out of memory after a long while, and System.gc() doesn't help, then you do have some sort of leak. Perhaps you have a stack or an arrayList or a hashMap or something of the sort that is slowly filling up? You mentioned the app needs to store some references initially - how many? Millions? If 10,000 records are being sorted in a lump, are you clearing out the temporary storage after each batch? Without seeing some code at this point, its hard to say - but the memory doesn't just get tired, its being filled with something :D

Every time I've run into a problem like this, it's a memory leak (or a cpu leak). It's important to know all the places where memory leaks are possible. The big obvious one is a collection that you keep adding to but never remove elements from.

But there are more insidious memory leaks in Java that are damn near impossible to track down unless you just know about them. For example, if you call "getGraphics" on a component and fail to call the "dispose" on that graphics when you're done. you get a leak. There are potential memory leaks in ImageIO, Runtime.exec, and anything else that does something native under the hood.

So what's a CPU leak? A list that you keep adding to and sorting can become a CPU leak (as well as a memory leak). Anything you do that keeps adding work the more you do it is potentially a CPU leak, and these can have the same symptoms as a memory leak.

OK.
Try jmap, with the -histo option. Possibly with ":live".
This will print out a histogram of objects in memory.
Since you're eating 6Gb in no time at all I'd guess it was the top few that would be your target...then figure out where they come from.

It's a lot more manual, but jmap won't eat your memory.

Don't forget to redirect the output to a file!

Thanks guys,
There was no leak but I just changed the way memory reference is released.

I have a big chunk of memory which I assign to a class variable.
Instead of making that variable null at the end of processing, I assign it to a different chunk of memory(and hence release reference to previously held memory chunk) in the next iteration.

Because of that, I had all these problems.
Thanks to PrintGCDetails!

I also stopped using eclipse to run the program and now use ant scripts to run directly from command prompt. The memory consumption is only 2.5 GB.

I am seeing a new problem now. :(

Java GC seems to be blocking my application forever.
After running for around 30-40 mins, my java application is blocked and Garbage collector seems to be running all the time.
All the threads of my application are blocked forever.
Memory consumption is far behind the max heap space and available system memory. So I am not sure why GC is so active.

I used debugging info from GC and it's saying my application ran for 1.9 seconds and it was stopped for 0.04 seconds.
But I am constantly watching the stack trace of my application. There is no progress at all. All threads are blocked at same point as it were for more than 15 mins.
I don't think there is any deadlock here. All the blocked threads are in synchronized() blocks or at Thread.join.
So they are all standard components.

Any help is much appreciated.

I would suggest you to change your default garbage collector by setting JVM parameters if you are using java 1.5 or above you can use Concurrent mark and sweep GC collector
(-XX:+UseConcMarkSweepGC) or Concurrent
Low pause collector

How do you know it's the garbage collector?

I have implemented a lot of (weird) algorithms, programs and applications in Java and have never found that the default garbage collector were inadequate. Always when an OutOfMemoryError was thrown at me there were some 'sticky' objects, i.e. those objects (or resources such as Socktets, open files etc.) where still reachable by my program somehow so the garbage collector couldn't and wouldn't collect them. Garbage collection is a well thought out technology and it's highly unlikely that there are still bugs in it. I do believe that the bug is somewhere in your own code.

kind regards,

Jos

Or, in other words, how do they know it's the garbage collector...;)

Quote:

---

Originally Posted by Tolls

Or, in other words, how do they know it's the garbage collector...;)

---

Clairvoyance and a bit of Voodoo, why ask?

kindest regards,

Jos ;-)

Quote:

---

Originally Posted by javaOrC

I am seeing a new problem now. :(

Java GC seems to be blocking my application forever.
After running for around 30-40 mins, my java application is blocked and Garbage collector seems to be running all the time.

---

I know it's tempting to blame some hidden magic component like the GC for your problems, but consider that java would be universally worthless if the garbage collector caused such problems. I've written high volume java apps that run for months on end and the GC works like a champ. Most likely, the activity you're seeing in the GC is a symptom not a cause.

There are some diagnostic tools in the jdk you can use to see what's going on in your threads. jconsole and jvisualvm are graphical tools that allow you to connect to a running java process and examine what's going on.

If you application is just stuck, it's probably deadlocked. If you've checked, doublechecked, and triple checked your code for potential deadlocks, you might be running up against an insidious deadlock. Does your application have a GUI? If it does, you have to be extremely careful with synchronized blocks because the AWT thread is off doing whatever it feels like, and you could end up causing it to block. You can tell if this is happening by using one of the aforementioned tools. If the AWT thread is stuck at one spot, you probably caused it to deadlock.

JVM tuning options
for Java 1.5 Tuning Garbage Collection with the 5.0 Java[tm] Virtual Machine
for 1.4 Tuning Garbage Collection with the 1.4.2 Java[tm] Virtual Machine

Except, as has been pointed out, it is quite likely this has nothing to do with the garbage collector.

Quote:

---

Originally Posted by Tolls

Except, as has been pointed out, it is quite likely this has nothing to do with the garbage collector.

---

It's been like that for centuries: if we don't understand how something works we're blaming it for faults; we've tortured and burned and dissected numerous people to death because we didn't understand them. Burn! Burn! ;-)

kind regards,

Jos

I have my pitchfork and flaming torch ready.
When do we storm Castle Garbage?