IO and Performance

[Cbani]

Hi Guys,
I am facing a performance related issue in my code. Below is the code snippet.
public static Map<Long, String> chunkFile(String fileName, String fromFileLocation, long interval, long parentId, String toFileLocation) throws Exception {

String thisLine;
String thisLine1 = GenericConstants.EMPTY_STRING;

long countRecord = 0;
BufferedReader brReader = null;
ChunkAndPriorityService chunkAndPriorityService = new ChunkAndPriorityServiceImpl();
long id = 0;
Map<Long, String> childFileMap = new HashMap<Long, String>();

try {

brReader = new BufferedReader(new FileReader(fromFileLocation+ GenericConstants.FILESEPARATOR +fileName));
countRecord = 0;
try{
fileName = fileName.substring(0, fileName.lastIndexOf(GenericConstants.DOT));
}catch(Exception ex){
fileExtension = GenericConstants.EMPTY_STRING;
ex.printStackTrace();
}
while ((thisLine = brReader.readLine()) != null) {
thisLine1 = thisLine1 + thisLine + GenericConstants.NEXT_LINE;
countRecord++;

if (countRecord == interval) {
id = chunkAndPriorityService.getChildSequence();
saveFileDetails(id, fileName, countRecord, parentId);
writeFile(toFileLocation, fileName, thisLine1, id);
childFileMap.put(id, fileName + id + fileExtension);
countRecord = 0;
thisLine1 = GenericConstants.EMPTY_STRING;
}
}
if (countRecord < interval) {
id = chunkAndPriorityService.getChildSequence();
saveFileDetails(id, fileName, countRecord, parentId);
writeFile(toFileLocation, fileName, thisLine1, id);
childFileMap.put(id, fileName + id + fileExtension);
}

} catch (IOException e) {
System.err.println("Error: " + e);
throw e;
} catch (Exception e) {
throw e;
}
return childFileMap;
}

below is the method which is responsible for writing the new files..
private static void writeFile(String filePath, String fileName, String fileContent, long id) throws IOException {
File newFile;
BufferedWriter out = null;
FileWriter fstream = null;

try {
newFile = new File(filePath + GenericConstants.FILESEPARATOR + fileName + id + fileExtension);
System.out.println("Writing file "+fileName+" @"+newFile);
newFile.createNewFile();
fstream = new FileWriter(newFile);
out = new BufferedWriter(fstream);
out.write(fileContent);
out.close();
} catch (IOException e) {
out.close();
e.printStackTrace();
throw e;
}

}
the method chunkFile() is responsible for reading a file (might be having 10,0000 records) and based on the interval (as passed in the method argument; lets say 10,000) it creates new files each having 10,000 records.

So, the issue here is

while ((thisLine = brReader.readLine()) != null) {
thisLine1 = thisLine1 + thisLine + GenericConstants.NEXT_LINE;
countRecord++;

if (countRecord == interval) {

the above code snippet takes a long time...
Can we do anmy kind of code tuning over here?
Any suggestions would be appreciated.

[Cbani]

any quick reply guys?

[JosAH]

any quick reply guys?

Make your thisLine1 a StringBuilder and append all your Strings to it; at the end convert your StringBuilder to a String.

kind regards,

Jos

[RamyaSivakanth]

Instead of building the string inside the method n number of times..u can build one time and call the method right.No need to build like this below

fromFileLocation+ GenericConstants.FILESEPARATOR +fileName...

you can build the string once and pass it inside this chunkFile method right?

[Cbani]

Make your thisLine1 a StringBuilder and append all your Strings to it; at the end convert your StringBuilder to a String.

the above solution is not working at all...as it goes out of memory

[Cbani]

Ramya,
The chunkFile method is called only once in its life cycle...so i dont think this an issue.
The issue is thisLine1(as it is holding lets say 10,000 records in it; where the actual file is 1,00000 and interval is 10,000) is using a lot of memory which is making the process too slow.
Do we have any API in java file system where i could read a file from one line to the other desired line; so that i could skip the thisLine1 code...

[RamyaSivakanth]

Just go thru this Java Examples - How do I read file contents to string using commons-io? whether it helps you.

Be specific with what you want exactly.

Iam not able to understand
"Do we have any API in java file system where i could read a file from one line to the other desired line; so that i could skip the thisLine1 code... "

[Cbani]

"Do we have any API in java file system where i could read a file from one line to the other desired line; so that i could skip the thisLine1 code... "

let me explain..

lets say i have a file of 1,000 lines. coz of some performance issues in reading the file line by line, i want some mechanism where i could read the file lets say from line 1 to line 100 then line101 to line200...and so on.I dont wanna read the 1st 1-100 lines line by line, instead i want some mechanism where i will specify the fromLineNumber to toLineNumber and the contents in between that would be read at once..

Hope this makes some sense

[RamyaSivakanth]

Hi,
I don't think any api in java like this.But,for reading the entire file we are having some 3rd party api like what i mentioned below.Just have a try and let me know.

-regards
Ramya

[JosAH]

Make your thisLine1 a StringBuilder and append all your Strings to it; at the end convert your StringBuilder to a String.

the above solution is not working at all...as it goes out of memory

You didn't implement it correctly then. Don't blame the suggestion, blame your incorrect implementation thereof instead.

kind regards,

Jos

[Cbani]

You didn't implement it correctly then. Don't blame the suggestion, blame your incorrect implementation thereof instead.

Cool down dude..
my intention was not to blame you rather to get a solution.Now below is the implementation as suggested by you

public static Map<Long, String> chunkFile(String fileName, String fromFileLocation, long interval, long parentId, String toFileLocation) throws Exception {

String thisLine;
StringBuilder thisLine1 = new StringBuilder(GenericConstants.EMPTY_STRING);

long countRecord = 0;
BufferedReader brReader = null;
ChunkAndPriorityService chunkAndPriorityService = new ChunkAndPriorityServiceImpl();
long id = 0;
Map<Long, String> childFileMap = new HashMap<Long, String>();

try {

brReader = new BufferedReader(new FileReader(fromFileLocation+ GenericConstants.FILESEPARATOR +fileName));
countRecord = 0;
try{
fileName = fileName.substring(0, fileName.lastIndexOf(GenericConstants.DOT));
}catch(Exception ex){
fileExtension = GenericConstants.EMPTY_STRING;
ex.printStackTrace();
}
while ((thisLine = brReader.readLine()) != null) {
//thisLine1 = thisLine1 + thisLine + GenericConstants.NEXT_LINE;
thisLine1.append(thisLine1);
thisLine1.append(thisLine);
thisLine1.append(GenericConstants.NEXT_LINE);
countRecord++;

if (countRecord == interval) {
id = chunkAndPriorityService.getChildSequence();
saveFileDetails(id, fileName, countRecord, parentId);
writeFile(toFileLocation, fileName, thisLine1.toString(), id);
childFileMap.put(id, fileName + id + fileExtension);
countRecord = 0;
thisLine1 = new StringBuilder(GenericConstants.EMPTY_STRING);
}
}
if ((countRecord != 0) && (countRecord < interval)) {
id = chunkAndPriorityService.getChildSequence();
saveFileDetails(id, fileName, countRecord, parentId);
writeFile(toFileLocation, fileName, thisLine1.toString(), id);
childFileMap.put(id, fileName + id + fileExtension);
}

} catch (IOException e) {
System.err.println("Error: " + e);
throw e;
} catch (Exception e) {
throw e;
}
return childFileMap;
}

private static void writeFile(String filePath, String fileName, String fileContent, long id) throws IOException {
File newFile;
BufferedWriter out = null;
FileWriter fstream = null;

try {
newFile = new File(filePath + GenericConstants.FILESEPARATOR + fileName + id + fileExtension);
System.out.println("Writing file "+fileName+" @"+newFile);
newFile.createNewFile();
fstream = new FileWriter(newFile);
out = new BufferedWriter(fstream);
out.write(fileContent);
out.close();
} catch (IOException e) {
out.close();
e.printStackTrace();
throw e;
}

}
Correct me if you find anything wrong in the implementation.
This is what is the exception log i am getting

java.lang.OutOfMemoryError: requested array is larger than heap
at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.
java:99)
at java.lang.StringBuilder.append(StringBuilder.java: 131)
at java.lang.StringBuilder.append(StringBuilder.java: 172)
at com.****.***.utility.ChunkFileUtil.chunkFile(Chunk FileUtil.java:1
00)
at com.****.****.adapters.ChunkAndPriorityServiceActi vatorBean.onMess
age(ChunkAndPriorityServiceActivatorBean.java:172)
at weblogic.ejb.container.internal.MDListener.execute (MDListener.java:46
6)
at weblogic.ejb.container.internal.MDListener.transac tionalOnMessage(MDL
istener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessa ge(MDListener.java:
327)
at weblogic.jms.client.JMSSession.onMessage(JMSSessio n.java:4072)
at weblogic.jms.client.JMSSession.execute(JMSSession. java:3964)
at weblogic.jms.client.JMSSession$UseForRunnable.run( JMSSession.java:449
0)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapte rImpl.run(SelfTunin
gWorkManagerImpl.java:464)
at weblogic.work.ExecuteThread.execute(ExecuteThread. java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java :172)

[JosAH]

				thisLine1.append(thisLine1);

Correct me if you find anything wrong in the implementation.
This is what is the exception log i am getting

java.lang.OutOfMemoryError: requested array is larger than heap
at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.
java:99)

That's what happens when you try to append a StringBuilder to itself; don't do that but think first before you attempt to code. The StringBuilder must be used to collect (append) the new content, there is no need to append the already present content.

kind regards,

Jos

[Cbani]

tahnks Josah,

really helpful....