Page 1 of 3 123 LastLast
Results 1 to 20 of 46
  1. #1
    javaOrC is offline Member
    Join Date
    Jan 2011
    Location
    NewYork
    Posts
    13
    Rep Power
    0

    Question Garbage Collector tuning

    Hi,
    I need some help to run my java application which processes millions of records. I do this in a batch fashion where I process around 10000 records in each batch.
    I have to create multiple objects and release references to most of them during each batch.

    Java Garbage collector(GC) doesn't seem to be actively freeing objects and hence I have my application getting very slow after 15-20 mins when memory is almost fully consumed.

    I tried tuning GC with: -XX:ExplicitGCInvokesConcurrent
    -XX:+UseAdapativeGCBoundary
    but they just postponed the delay problem.

    I am sure java should have some option to handle this issue.
    Can somebody suggest any idea?

    thanks,
    Dilip

  2. #2
    lovelesh is offline Senior Member
    Join Date
    Nov 2010
    Location
    Delhi
    Posts
    135
    Blog Entries
    1
    Rep Power
    0

    Post

    This doesn't seems to be Garbage Collection problem alone.

    I sense some memory leak in your program. Review your code for any references not freed up. You can use any profiler to figure out the exact object which is not getting freed up. As you might be using threads to process each record, several threads eventually consume your whole memory.

  3. #3
    quad64bit's Avatar
    quad64bit is offline Moderator
    Join Date
    Jul 2009
    Location
    VA
    Posts
    1,323
    Rep Power
    7

    Default

    Just out of curiosity, if you put System.GC() into your loop, does memory still fill up after 15-20 min?

  4. #4
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,091
    Rep Power
    20

    Default

    Quote Originally Posted by lovelesh View Post
    This doesn't seems to be Garbage Collection problem alone.

    I sense some memory leak in your program. Review your code for any references not freed up. You can use any profiler to figure out the exact object which is not getting freed up. As you might be using threads to process each record, several threads eventually consume your whole memory.
    This.
    If the memory is slowly being consumed like this then you have a leak.

    First get a heap dump from your app as it's slowing down (so after 15 minutes or so) and then stick that in an analyser, like Eclipse MAT. This should highlight the objects that are the likely root cause of your problem. From there you should be able to figure out why they aren't being cleared out.

    We can make guesses here, but until you've done that step that's all they'll be...guesses.

  5. #5
    javaOrC is offline Member
    Join Date
    Jan 2011
    Location
    NewYork
    Posts
    13
    Rep Power
    0

    Default

    I checked with the mem dumps.
    The leak report says no leak.

    I also tried using system.gc very frequently, but it doesn't seem to help either.

    I got an out of memory error after 20-30 mins and the dump size was 6.1 GB.
    I am unable to load it in eclipse as MAT tool in eclipse says: Error: Java heap space.

    From the GCDetails printed, I see that GC is blocking the application for around 9-10 seconds very often after 15-20 mins of starting the program.
    My program is expected to hold references to some memory objects initially.
    So initially I expect GC to free fewer objects and later on GC should be able to free huge chunk of short lived objects.

    The number of objects I hold reference to is finite and much lesser than what my physical memory(8GB) can accommodate.

    Any advice?

  6. #6
    quad64bit's Avatar
    quad64bit is offline Moderator
    Join Date
    Jul 2009
    Location
    VA
    Posts
    1,323
    Rep Power
    7

    Default

    Well, as far as I know, java can still only use 2GB max per instance of a VM. You might try adding -Xmx2048m -Xms1024m or something similar to your launch arguments to see if it helps (the defaults are much much lower). However, if you are only running out of memory after a long while, and System.gc() doesn't help, then you do have some sort of leak. Perhaps you have a stack or an arrayList or a hashMap or something of the sort that is slowly filling up? You mentioned the app needs to store some references initially - how many? Millions? If 10,000 records are being sorted in a lump, are you clearing out the temporary storage after each batch? Without seeing some code at this point, its hard to say - but the memory doesn't just get tired, its being filled with something :D

  7. #7
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    Every time I've run into a problem like this, it's a memory leak (or a cpu leak). It's important to know all the places where memory leaks are possible. The big obvious one is a collection that you keep adding to but never remove elements from.

    But there are more insidious memory leaks in Java that are damn near impossible to track down unless you just know about them. For example, if you call "getGraphics" on a component and fail to call the "dispose" on that graphics when you're done. you get a leak. There are potential memory leaks in ImageIO, Runtime.exec, and anything else that does something native under the hood.

    So what's a CPU leak? A list that you keep adding to and sorting can become a CPU leak (as well as a memory leak). Anything you do that keeps adding work the more you do it is potentially a CPU leak, and these can have the same symptoms as a memory leak.

  8. #8
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,091
    Rep Power
    20

    Default

    OK.
    Try jmap, with the -histo option. Possibly with ":live".
    This will print out a histogram of objects in memory.
    Since you're eating 6Gb in no time at all I'd guess it was the top few that would be your target...then figure out where they come from.

    It's a lot more manual, but jmap won't eat your memory.

    Don't forget to redirect the output to a file!

  9. #9
    javaOrC is offline Member
    Join Date
    Jan 2011
    Location
    NewYork
    Posts
    13
    Rep Power
    0

    Default

    Thanks guys,
    There was no leak but I just changed the way memory reference is released.

    I have a big chunk of memory which I assign to a class variable.
    Instead of making that variable null at the end of processing, I assign it to a different chunk of memory(and hence release reference to previously held memory chunk) in the next iteration.

    Because of that, I had all these problems.
    Thanks to PrintGCDetails!

    I also stopped using eclipse to run the program and now use ant scripts to run directly from command prompt. The memory consumption is only 2.5 GB.

  10. #10
    javaOrC is offline Member
    Join Date
    Jan 2011
    Location
    NewYork
    Posts
    13
    Rep Power
    0

    Default Java GC running forever

    I am seeing a new problem now. :(

    Java GC seems to be blocking my application forever.
    After running for around 30-40 mins, my java application is blocked and Garbage collector seems to be running all the time.
    All the threads of my application are blocked forever.
    Memory consumption is far behind the max heap space and available system memory. So I am not sure why GC is so active.

    I used debugging info from GC and it's saying my application ran for 1.9 seconds and it was stopped for 0.04 seconds.
    But I am constantly watching the stack trace of my application. There is no progress at all. All threads are blocked at same point as it were for more than 15 mins.
    I don't think there is any deadlock here. All the blocked threads are in synchronized() blocks or at Thread.join.
    So they are all standard components.

    Any help is much appreciated.

  11. #11
    pandeyvic is offline Member
    Join Date
    Jan 2011
    Posts
    3
    Rep Power
    0

    Default

    I would suggest you to change your default garbage collector by setting JVM parameters if you are using java 1.5 or above you can use Concurrent mark and sweep GC collector
    (-XX:+UseConcMarkSweepGC) or Concurrent
    Low pause collector

  12. #12
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,091
    Rep Power
    20

    Default

    How do you know it's the garbage collector?

  13. #13
    JosAH's Avatar
    JosAH is offline Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,651
    Blog Entries
    7
    Rep Power
    21

    Default

    I have implemented a lot of (weird) algorithms, programs and applications in Java and have never found that the default garbage collector were inadequate. Always when an OutOfMemoryError was thrown at me there were some 'sticky' objects, i.e. those objects (or resources such as Socktets, open files etc.) where still reachable by my program somehow so the garbage collector couldn't and wouldn't collect them. Garbage collection is a well thought out technology and it's highly unlikely that there are still bugs in it. I do believe that the bug is somewhere in your own code.

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  14. #14
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,091
    Rep Power
    20

    Default

    Or, in other words, how do they know it's the garbage collector...;)

  15. #15
    JosAH's Avatar
    JosAH is offline Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,651
    Blog Entries
    7
    Rep Power
    21

    Default

    Quote Originally Posted by Tolls View Post
    Or, in other words, how do they know it's the garbage collector...;)
    Clairvoyance and a bit of Voodoo, why ask?

    kindest regards,

    Jos ;-)
    cenosillicaphobia: the fear for an empty beer glass

  16. #16
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    Quote Originally Posted by javaOrC View Post
    I am seeing a new problem now. :(

    Java GC seems to be blocking my application forever.
    After running for around 30-40 mins, my java application is blocked and Garbage collector seems to be running all the time.
    I know it's tempting to blame some hidden magic component like the GC for your problems, but consider that java would be universally worthless if the garbage collector caused such problems. I've written high volume java apps that run for months on end and the GC works like a champ. Most likely, the activity you're seeing in the GC is a symptom not a cause.

    There are some diagnostic tools in the jdk you can use to see what's going on in your threads. jconsole and jvisualvm are graphical tools that allow you to connect to a running java process and examine what's going on.

    If you application is just stuck, it's probably deadlocked. If you've checked, doublechecked, and triple checked your code for potential deadlocks, you might be running up against an insidious deadlock. Does your application have a GUI? If it does, you have to be extremely careful with synchronized blocks because the AWT thread is off doing whatever it feels like, and you could end up causing it to block. You can tell if this is happening by using one of the aforementioned tools. If the AWT thread is stuck at one spot, you probably caused it to deadlock.

  17. #17
    pandeyvic is offline Member
    Join Date
    Jan 2011
    Posts
    3
    Rep Power
    0

  18. #18
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,091
    Rep Power
    20

    Default

    Except, as has been pointed out, it is quite likely this has nothing to do with the garbage collector.

  19. #19
    JosAH's Avatar
    JosAH is offline Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,651
    Blog Entries
    7
    Rep Power
    21

    Default

    Quote Originally Posted by Tolls View Post
    Except, as has been pointed out, it is quite likely this has nothing to do with the garbage collector.
    It's been like that for centuries: if we don't understand how something works we're blaming it for faults; we've tortured and burned and dissected numerous people to death because we didn't understand them. Burn! Burn! ;-)

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  20. #20
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,091
    Rep Power
    20

    Default

    I have my pitchfork and flaming torch ready.
    When do we storm Castle Garbage?

Page 1 of 3 123 LastLast

Similar Threads

  1. In defense of the Garbage Collector
    By Katanagas in forum Advanced Java
    Replies: 2
    Last Post: 10-25-2010, 06:40 PM
  2. Garbage Collector Doubt
    By CristOm in forum New To Java
    Replies: 9
    Last Post: 05-04-2010, 11:26 AM
  3. Q about Garbage Collector
    By m00nchile in forum New To Java
    Replies: 4
    Last Post: 02-05-2010, 05:57 AM
  4. Garbage Collector and finalize()
    By arefeh in forum New To Java
    Replies: 5
    Last Post: 01-09-2010, 09:04 PM
  5. Garbage collector and its impacts
    By RadhaJayalakshmi in forum Advanced Java
    Replies: 1
    Last Post: 07-23-2008, 11:56 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •