trying to improve code to avoid java.lang.OutOfMemory error

[bdyarem]

Hello,

I am a newcomer to the forums here and Java novice. I have searched around quite a bit and will continue to do so, but thought I'd put this question to the user community in hopes someone can point out an obvious mistake I may be making...

I have a java class that executes a series of file loads, and as these file loads are executed the program uses a large amount of heap memory, eventually causing a failure with the java.lang.OutOfMemoryError. I can increase the JVM memory with -Xmx1024m (about the largest I can go on the machine running the program), and in most cases this will allow the program to run to completion. However, when given a large input, 1024m is still proving to be insufficient, and I'm pretty sure my code could use a few obvious improvements to allow for less memory consumption.

Using JProbe profiler, it appears my program consumes (and doesn't release) a large amount of memory during the following statement, which writes to a CSV file. ps2 is a PrintStream object that I am declaring with the autoflush = true option (which isn't seeming to help). Basically this nasty code is writing my CSV formatted file, and this statement can be iterated in upwards of 40K times. I've also tried explicitly nulling the ps2 object and forcing a garbage collection with gc() after all rows have been written, but isn't helping either. Any suggestions?


for(x=0;x < myQuery.getResultSize();x++)
{
ps2.println("\"" + myInfoObject.getFiles().myFilePathMethod() + "\"" + "," +
"\"" + ((IFile)myInfoObject.getFiles().get(0)).getName() + "\"" + "," +
"\"" + ((IFile)myInfoObject.getFiles().get(0)).getSize() + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN1") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN2") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN3") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN4") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN5") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN6") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN7") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN8") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN9") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN10") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN11") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN12") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN13") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN14") + "\"");

}

[Norm]

series of file loads

What does this mean? Does each file get its own storage or do you reuse the storage the file loads into?
If each file loads to its own storage, then you'll run out of memory.

[bdyarem]

There is a series of ~40 csv files that are written to, all of which use similiar code to what I've pasted into the original post. They are all currently using their own "ps#" printStream object for storage. I will definately re-write this portion so they all re-use the same printStream. I suppose after each CSV file has been written, I can just ps# = null; and maybe force a garbage collection with gc()? Hopefully this will resolve the memory issue - thanks for your prompt reply!

[Norm]

I was assuming that "file loads" meant input files.
What is the logic flow in the program?
Read a file, write out a file done in a loop, one file at a time
Or read all the input files and then write out files

[bdyarem]

I was assuming that "file loads" meant input files.
What is the logic flow in the program?
Read a file, write out a file done in a loop, one file at a time
Or read all the input files and then write out files

One at a time, the program reads information from a database table, writes this information to a CSV file (writing done in loop as shown). The entire process is carried out sequentially, about 40 times for queries / CSVs of various sizes.

[Eku]

Do you mean you are processing large volume of files?
That is really not a good idea. Ive tried processing 27KB of a flat file and I got that same error. Of cource each line of the file has a lot of database validation and insertion. What i did is i cut the file manually. I hope someone here had a solution to that. My machine cannot handle that large volume of files.

I hope someone here can give us a solution to our problems ^_^

My suggestion is, process them one at a time. or have a limit like 5KB per execution. Lets wait for the others to reply on this thread

[Eranga]

What I suggest is, as Eku says, use the files in smaller size, and process one at a time. But there can be lots of issues when splitting files in to smaller parts. May there can be data lost. So the file size should handle when they are creating.

And also, in such a huge application I do a tricky thing. Regularly I check the application memory usage. Depend on the usage and the pre-defined memory limit run the garbage collector. So far I'm success with that.

[DevzAbhi]

Try using string buffer instead of string

[Eranga]

Try using string buffer instead of string

What's the reason for that. Are you talking about String constant area? Actually those things depends on the implementation of your application.

[DevzAbhi]

Actually every time he is concanating the string a new String object is created

[Eranga]

Ah, that's a bad practice. But I don't think string object effect a lot for memory leak in an application. Class object reference and stuff make a real sense.

[skaspersen]

1) Use StringBuilder to create your strings, or if you are using threads StringBuffer.
2)Don't keep resources open if you are not using them. Create the PrintStream object before writing to it and close it once you are done. It is good practice to do this in a finally block.

PrintStream ps = null;
try {
	ps = new PrintStream("filename");
	//Work with ps
} catch (SomeException e) {
	// Handle Exception
} catch ....
} finally {
	if (ps != null) {
		ps.close();
	}
}

3)If possible use limmit in your DB query,
This would mean executing a query to get the number of rows in the table first. Also close your query and resultset once done


eg

open ps
1st Query "Select * from TABLE limmit 0,1000". 
Add Data to file
Close Query Object
2nd Query "Select * from TABLE limmit 1001,2000". 
Add Data to file
Close Query Object
...
...
...
ps.close();

HTH
Stephen

[Norm]

To track down your problem I suggest you use the freeMemory and totalMemory methods to display memory usage.
Break down your too long statement into single statements (ie each get() in a separate statement and each String to a separate variable) and use println() with the memory display methods to show how memory usage changes from one statement to another statement.

[Eranga]

Yes I second your suggestions Norm. In simple word dealing with the garbage collector manually, rather letting on JVM, is a good idea. In most of the industrial applications I keep records of those details when a customer report a bug. :) Those details are really important to us keep my application in safer way.

[bdyarem]

1) Use StringBuilder to create your strings, or if you are using threads StringBuffer.
2)Don't keep resources open if you are not using them. Create the PrintStream object before writing to it and close it once you are done. It is good practice to do this in a finally block.

PrintStream ps = null;
try {
	ps = new PrintStream("filename");
	//Work with ps
} catch (SomeException e) {
	// Handle Exception
} catch ....
} finally {
	if (ps != null) {
		ps.close();
	}
}

3)If possible use limmit in your DB query,
This would mean executing a query to get the number of rows in the table first. Also close your query and resultset once done


eg

open ps
1st Query "Select * from TABLE limmit 0,1000". 
Add Data to file
Close Query Object
2nd Query "Select * from TABLE limmit 1001,2000". 
Add Data to file
Close Query Object
...
...
...
ps.close();

HTH
Stephen

Thank you all for your suggestions. I appear to have fixed my memory usage issue, and I'm afraid the solution was actually quite simple: 1) I simply had to null out each query result object after I was through using it, and 2) I re-used 1 printStream object for all of the CSV file writing. Thanks again!

[Eranga]

Reusing resources is the most effective way in developing. But you have to be careful when use of again, the previous values should be cleared.

[SecretCode]

\But I don't think string object effect a lot for memory leak in an application.

Although the OP's problem is solved, for other readers it might still be worth considering the issues of Strings. I haven't done any such memory-intensive programs in Java, but certainly in other languages that reallocate strings you can use vast amounts of memory very quickly.