Results 1 to 17 of 17
  1. #1
    bdyarem is offline Member
    Join Date
    Jul 2008
    Posts
    4
    Rep Power
    0

    Question trying to improve code to avoid java.lang.OutOfMemory error

    Hello,

    I am a newcomer to the forums here and Java novice. I have searched around quite a bit and will continue to do so, but thought I'd put this question to the user community in hopes someone can point out an obvious mistake I may be making...

    I have a java class that executes a series of file loads, and as these file loads are executed the program uses a large amount of heap memory, eventually causing a failure with the java.lang.OutOfMemoryError. I can increase the JVM memory with -Xmx1024m (about the largest I can go on the machine running the program), and in most cases this will allow the program to run to completion. However, when given a large input, 1024m is still proving to be insufficient, and I'm pretty sure my code could use a few obvious improvements to allow for less memory consumption.

    Using JProbe profiler, it appears my program consumes (and doesn't release) a large amount of memory during the following statement, which writes to a CSV file. ps2 is a PrintStream object that I am declaring with the autoflush = true option (which isn't seeming to help). Basically this nasty code is writing my CSV formatted file, and this statement can be iterated in upwards of 40K times. I've also tried explicitly nulling the ps2 object and forcing a garbage collection with gc() after all rows have been written, but isn't helping either. Any suggestions?


    for(x=0;x < myQuery.getResultSize();x++)
    {
    ps2.println("\"" + myInfoObject.getFiles().myFilePathMethod() + "\"" + "," +
    "\"" + ((IFile)myInfoObject.getFiles().get(0)).getName() + "\"" + "," +
    "\"" + ((IFile)myInfoObject.getFiles().get(0)).getSize() + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN1") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN2") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN3") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN4") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN5") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN6") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN7") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN8") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN9") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN10") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN11") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN12") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN13") + "\"" + "," +
    "\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN14") + "\"");

    }

  2. #2
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,409
    Rep Power
    25

    Default

    series of file loads
    What does this mean? Does each file get its own storage or do you reuse the storage the file loads into?
    If each file loads to its own storage, then you'll run out of memory.

  3. #3
    bdyarem is offline Member
    Join Date
    Jul 2008
    Posts
    4
    Rep Power
    0

    Default

    There is a series of ~40 csv files that are written to, all of which use similiar code to what I've pasted into the original post. They are all currently using their own "ps#" printStream object for storage. I will definately re-write this portion so they all re-use the same printStream. I suppose after each CSV file has been written, I can just ps# = null; and maybe force a garbage collection with gc()? Hopefully this will resolve the memory issue - thanks for your prompt reply!

  4. #4
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,409
    Rep Power
    25

    Default

    I was assuming that "file loads" meant input files.
    What is the logic flow in the program?
    Read a file, write out a file done in a loop, one file at a time
    Or read all the input files and then write out files

  5. #5
    bdyarem is offline Member
    Join Date
    Jul 2008
    Posts
    4
    Rep Power
    0

    Default

    Quote Originally Posted by Norm View Post
    I was assuming that "file loads" meant input files.
    What is the logic flow in the program?
    Read a file, write out a file done in a loop, one file at a time
    Or read all the input files and then write out files
    One at a time, the program reads information from a database table, writes this information to a CSV file (writing done in loop as shown). The entire process is carried out sequentially, about 40 times for queries / CSVs of various sizes.

  6. #6
    Eku
    Eku is offline Senior Member
    Join Date
    May 2008
    Location
    Makati, Philippines
    Posts
    234
    Rep Power
    7

    Default

    Do you mean you are processing large volume of files?
    That is really not a good idea. Ive tried processing 27KB of a flat file and I got that same error. Of cource each line of the file has a lot of database validation and insertion. What i did is i cut the file manually. I hope someone here had a solution to that. My machine cannot handle that large volume of files.

    I hope someone here can give us a solution to our problems ^_^

    My suggestion is, process them one at a time. or have a limit like 5KB per execution. Lets wait for the others to reply on this thread
    Mind only knows what lies near the heart, it alone sees the depth of the soul.

  7. #7
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

    Default

    What I suggest is, as Eku says, use the files in smaller size, and process one at a time. But there can be lots of issues when splitting files in to smaller parts. May there can be data lost. So the file size should handle when they are creating.

    And also, in such a huge application I do a tricky thing. Regularly I check the application memory usage. Depend on the usage and the pre-defined memory limit run the garbage collector. So far I'm success with that.

  8. #8
    DevzAbhi's Avatar
    DevzAbhi is offline Member
    Join Date
    Jul 2008
    Posts
    67
    Rep Power
    0

    Default

    Try using string buffer instead of string

  9. #9
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

    Default

    Quote Originally Posted by DevzAbhi View Post
    Try using string buffer instead of string
    What's the reason for that. Are you talking about String constant area? Actually those things depends on the implementation of your application.

  10. #10
    DevzAbhi's Avatar
    DevzAbhi is offline Member
    Join Date
    Jul 2008
    Posts
    67
    Rep Power
    0

    Default

    Actually every time he is concanating the string a new String object is created

  11. #11
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

    Default

    Ah, that's a bad practice. But I don't think string object effect a lot for memory leak in an application. Class object reference and stuff make a real sense.

  12. #12
    skaspersen is offline Member
    Join Date
    Jul 2008
    Posts
    31
    Rep Power
    0

    Default

    1) Use StringBuilder to create your strings, or if you are using threads StringBuffer.
    2)Don't keep resources open if you are not using them. Create the PrintStream object before writing to it and close it once you are done. It is good practice to do this in a finally block.

    Java Code:
    PrintStream ps = null;
    try {
    	ps = new PrintStream("filename");
    	//Work with ps
    } catch (SomeException e) {
    	// Handle Exception
    } catch ....
    } finally {
    	if (ps != null) {
    		ps.close();
    	}
    }
    3)If possible use limmit in your DB query,
    This would mean executing a query to get the number of rows in the table first. Also close your query and resultset once done


    eg
    Java Code:
    open ps
    1st Query "Select * from TABLE limmit 0,1000". 
    Add Data to file
    Close Query Object
    2nd Query "Select * from TABLE limmit 1001,2000". 
    Add Data to file
    Close Query Object
    ...
    ...
    ...
    ps.close();
    HTH
    Stephen

  13. #13
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,409
    Rep Power
    25

    Default

    To track down your problem I suggest you use the freeMemory and totalMemory methods to display memory usage.
    Break down your too long statement into single statements (ie each get() in a separate statement and each String to a separate variable) and use println() with the memory display methods to show how memory usage changes from one statement to another statement.

  14. #14
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

    Default

    Yes I second your suggestions Norm. In simple word dealing with the garbage collector manually, rather letting on JVM, is a good idea. In most of the industrial applications I keep records of those details when a customer report a bug. :) Those details are really important to us keep my application in safer way.

  15. #15
    bdyarem is offline Member
    Join Date
    Jul 2008
    Posts
    4
    Rep Power
    0

    Red face

    Quote Originally Posted by skaspersen View Post
    1) Use StringBuilder to create your strings, or if you are using threads StringBuffer.
    2)Don't keep resources open if you are not using them. Create the PrintStream object before writing to it and close it once you are done. It is good practice to do this in a finally block.

    Java Code:
    PrintStream ps = null;
    try {
    	ps = new PrintStream("filename");
    	//Work with ps
    } catch (SomeException e) {
    	// Handle Exception
    } catch ....
    } finally {
    	if (ps != null) {
    		ps.close();
    	}
    }
    3)If possible use limmit in your DB query,
    This would mean executing a query to get the number of rows in the table first. Also close your query and resultset once done


    eg
    Java Code:
    open ps
    1st Query "Select * from TABLE limmit 0,1000". 
    Add Data to file
    Close Query Object
    2nd Query "Select * from TABLE limmit 1001,2000". 
    Add Data to file
    Close Query Object
    ...
    ...
    ...
    ps.close();
    HTH
    Stephen
    Thank you all for your suggestions. I appear to have fixed my memory usage issue, and I'm afraid the solution was actually quite simple: 1) I simply had to null out each query result object after I was through using it, and 2) I re-used 1 printStream object for all of the CSV file writing. Thanks again!

  16. #16
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

    Default

    Reusing resources is the most effective way in developing. But you have to be careful when use of again, the previous values should be cleared.

  17. #17
    SecretCode is offline Member
    Join Date
    Jun 2008
    Location
    South Africa
    Posts
    6
    Rep Power
    0

    Default

    Quote Originally Posted by Eranga View Post
    \But I don't think string object effect a lot for memory leak in an application.
    Although the OP's problem is solved, for other readers it might still be worth considering the issues of Strings. I haven't done any such memory-intensive programs in Java, but certainly in other languages that reallocate strings you can use vast amounts of memory very quickly.

Similar Threads

  1. Replies: 2
    Last Post: 09-18-2008, 07:30 AM
  2. Error: cannot be applied to (java.lang.String)
    By carl in forum New To Java
    Replies: 1
    Last Post: 08-05-2007, 06:33 AM
  3. Replies: 1
    Last Post: 07-31-2007, 11:47 PM
  4. Replies: 1
    Last Post: 07-31-2007, 05:52 AM
  5. Replies: 2
    Last Post: 07-02-2007, 02:29 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •