Results 1 to 14 of 14
  1. #1
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    hi

    I'm running a java application that is supposed to process a batch of files through some decisioning module. I need to process the batch for about 50 hrs. The problem I'm facing is that the process runs fine for about an hour and then starts to idle. So, I did this - I run the JVM for one hour and then shut it down, restart the JVM after 30 mins, but still for some reason the second run is taking almost 4-5 hrs. to do what the first run does in 1 hr. Any help or insights would be greatly appreciated.

    I am running this on a 64-bit windows r2 server, 2 intel quad core processors(2.53 GHz), 24 GB RAM. Java version is 1.6.0_22(64-bit), memory allotted to the application is - heap(16 GB) and PermGen(2GB).

    Regards PJ

  2. #2
    doWhile is offline Moderator
    Join Date
    Jul 2010
    Location
    California
    Posts
    1,642
    Rep Power
    7

    Default

    There can be any number of problems...what is the code you running?

  3. #3
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    the code is just reading a bunch of files from a folder and sending a substring of the content to an external module, the module returns a response string which my code reads and writes back to the file system. It is a multithreaded process.

    As I said it works perfectly for an hour or so, the surprising part is (which I am unable to comprehend) - if I run it for one hour, shut down the JVM and start the JVM again after a while it seems to be slower the next time, almost seems to me like the CPU is getting tired.

    The external module (which runs on the same JVM) does have some native code which I suspect might be a cause.

  4. #4
    doWhile is offline Moderator
    Join Date
    Jul 2010
    Location
    California
    Posts
    1,642
    Rep Power
    7

    Default

    How long does the 'external' process take? If you call it from multiple threads, and that process takes a while and calls native processes, then those processes may still be active if you stop the JVM and thus tie up the processors when you restart the app. Can your threads be deadlocking? You could pull out the code calling the native processes and insert some dummy code to see if it is your app that could be deadlocking. If not, then it might be the native process.

  5. #5
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    i don't think there is a thread deadlock issue, i used the same setup in another process, over there instead of invoking the external module i was just modifying the strings in the java code.
    if it is a problem with the external module, can i anyway release the resources it is holding on to. I cannot modify the module contents as such but i do have the flexibility of setting everything back to square one - only if i knew a way to do so.

  6. #6
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    @doWhile
    i've noticed something.
    after shutting down the JVM, if i delete all the files that i wrote to the file system then the RAM usage comes back down to normal.
    Does that indicate anything and can I do something about it.

    this is my file i/o code which looks fine to me

    File outFile = new File("D:\\testOutput\\Output"+i+_fileName);
    FileWriter out = new FileWriter(outFile);
    out.write(string); //string is the response returned from the external module
    out.close();

  7. #7
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default

    I've also tried BufferedWriter but still have the same problem

    FileWriter fstream = new FileWriter("D:\\testOutput\\Output"+i+_fileName);
    BufferedWriter out = new BufferedWriter(fstream);
    out.write(string);
    out.close();

  8. #8
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    There really isn't enough detail to diagnose this, but here are some possibilities:

    - You are running into OS caching issues. I've seen this on linux and in spite of all the claims that it can't hurt you, it certainly can. The only solution I've found that works when running into this problem is to use direct IO (there's probably a java api out there somewhere that does this)

    - You are running out of real memory and so the disk virtual memory is being used. You can bring up task manager and examine the performance to see if that's the issue.

    - Your decisioning module is has to work harder over time. This can happen, for example, if it has a decision tree it has to traverse, and the tree keeps growing unbounded.

    - You are running out of some system resource, for example file handles. If for some reason exceptions are being thrown such that the 'out.close' statements are not being reached, then you will run out of file handles. Make sure you are not ignoring exceptions. If you are, odds are they are being thrown and you just don't know about it.

    You mentioned that you are sending strings to an external module. By chance, are you using Runtime.exec to launch that module? One common mistake is to fail to properly clean up after runtime.exec. You need to explicitly close the input stream, the error stream, and the output stream, even if you never use them, and you must also call 'dispose' on the Process object. That's 4 cleanup calls for every Runtime.exec call, and you need to absoluetly guarantee they will be called even in the event of exceptions.

    Java Code:
    Process p = null;
    try {
      p = Runtime.getRuntime.exec(...whatever...);
    
    } catch (...handle exceptions appropriately...) {
    
    } finally {
    
    
    try {p.getInputStream().close();} catch (Throwable t) {....}
    try {p.getErrorStream().close();} catch (Throwable t) {...}
    try {p.getOutputStream().close();} catch (Throwable t) {...}
    try {p.destroy(); } catch (Throwable t) {...log something}
    
    }
    }

  9. #9
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    i'm doing this on a windows server box.

    external module seems to be fine, if i execute the whole process and just bypass the file write to the hard disk then everything looks perfect. can't see why file write is holding memory even after jvm shutdown.
    my out.close() is in a finally block so it should get executed regardless of exceptions.

    i'm not using any Runtime.exec() calls

  10. #10
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    Ok, well it does sound like a caching issue then. The OS will cache files, which uses up RAM, and they will tend to remain cached until the file is either deleted, or the RAM is needed for something else. The problem often is that when the RAM is needed, the OS will then write to disk to free up cache memory. This can be very time consuming.

    Are the files that are getting written really needed long term? If not, you could more actively manage them by deleting them as soon as you no longer need them.

  11. #11
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    My problem is i need to process 50 million files and then another process will pick it up. Only then I can delete the files.
    I start seeing performance drops after processing 1 million files.

    I can move all the files from the hard drive to a mapped network drive, will the OS release the cache then? i can transfer the files after processing 1 million records.

  12. #12
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    If it is a cahing issue, then moving the files might not solve the problem, since they will probably still remain cached. This sounds like a situation where the right answer may be to throw hardware at it - greatly increase RAM and use a high speed RAID for storage.

  13. #13
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    i'll try out a few things. deleting the files seems to clear out the cache, i'll see if copying to network drive and deleting the original set helps or not.

  14. #14
    pjpr is offline Member
    Join Date
    Jan 2011
    Posts
    9
    Rep Power
    0

    Default Strange JVM behaviour

    it seems that writing to a mapped network drive instead of writing on the machine's disk drive doesn't cause this problem. Does this sound logical?
    I'll test it a bit further to confirm this. Thanks for the advice toadaly.

Similar Threads

  1. Strange eclipse behaviour with new version
    By KingOfLions in forum Eclipse
    Replies: 0
    Last Post: 09-10-2009, 01:42 PM
  2. strange JFace dialog behaviour
    By crni.adjah in forum SWT / JFace
    Replies: 0
    Last Post: 09-03-2009, 04:38 AM
  3. Strange behaviour in serialization
    By Wolverine in forum Networking
    Replies: 0
    Last Post: 05-23-2009, 12:03 PM
  4. AffinedTransform strange behaviour
    By Echilon in forum AWT / Swing
    Replies: 3
    Last Post: 12-11-2008, 09:58 AM
  5. Strange behaviour in swing
    By cbalu in forum AWT / Swing
    Replies: 1
    Last Post: 05-23-2008, 09:23 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •