Results 1 to 13 of 13
  1. #1
    Cbani is offline Member
    Join Date
    Jan 2010
    Posts
    90
    Rep Power
    0

    Default IO and Performance

    Hi Guys,
    I am facing a performance related issue in my code. Below is the code snippet.
    public static Map<Long, String> chunkFile(String fileName, String fromFileLocation, long interval, long parentId, String toFileLocation) throws Exception {

    String thisLine;
    String thisLine1 = GenericConstants.EMPTY_STRING;

    long countRecord = 0;
    BufferedReader brReader = null;
    ChunkAndPriorityService chunkAndPriorityService = new ChunkAndPriorityServiceImpl();
    long id = 0;
    Map<Long, String> childFileMap = new HashMap<Long, String>();

    try {

    brReader = new BufferedReader(new FileReader(fromFileLocation+ GenericConstants.FILESEPARATOR +fileName));
    countRecord = 0;
    try{
    fileName = fileName.substring(0, fileName.lastIndexOf(GenericConstants.DOT));
    }catch(Exception ex){
    fileExtension = GenericConstants.EMPTY_STRING;
    ex.printStackTrace();
    }
    while ((thisLine = brReader.readLine()) != null) {
    thisLine1 = thisLine1 + thisLine + GenericConstants.NEXT_LINE;
    countRecord++;

    if (countRecord == interval) {
    id = chunkAndPriorityService.getChildSequence();
    saveFileDetails(id, fileName, countRecord, parentId);
    writeFile(toFileLocation, fileName, thisLine1, id);
    childFileMap.put(id, fileName + id + fileExtension);
    countRecord = 0;
    thisLine1 = GenericConstants.EMPTY_STRING;
    }
    }
    if (countRecord < interval) {
    id = chunkAndPriorityService.getChildSequence();
    saveFileDetails(id, fileName, countRecord, parentId);
    writeFile(toFileLocation, fileName, thisLine1, id);
    childFileMap.put(id, fileName + id + fileExtension);
    }

    } catch (IOException e) {
    System.err.println("Error: " + e);
    throw e;
    } catch (Exception e) {
    throw e;
    }
    return childFileMap;
    }

    below is the method which is responsible for writing the new files..
    private static void writeFile(String filePath, String fileName, String fileContent, long id) throws IOException {
    File newFile;
    BufferedWriter out = null;
    FileWriter fstream = null;

    try {
    newFile = new File(filePath + GenericConstants.FILESEPARATOR + fileName + id + fileExtension);
    System.out.println("Writing file "+fileName+" @"+newFile);
    newFile.createNewFile();
    fstream = new FileWriter(newFile);
    out = new BufferedWriter(fstream);
    out.write(fileContent);
    out.close();
    } catch (IOException e) {
    out.close();
    e.printStackTrace();
    throw e;
    }

    }
    the method chunkFile() is responsible for reading a file (might be having 10,0000 records) and based on the interval (as passed in the method argument; lets say 10,000) it creates new files each having 10,000 records.

    So, the issue here is

    while ((thisLine = brReader.readLine()) != null) {
    thisLine1 = thisLine1 + thisLine + GenericConstants.NEXT_LINE;
    countRecord++;

    if (countRecord == interval) {

    the above code snippet takes a long time...
    Can we do anmy kind of code tuning over here?
    Any suggestions would be appreciated.

  2. #2
    Cbani is offline Member
    Join Date
    Jan 2010
    Posts
    90
    Rep Power
    0

    Default

    any quick reply guys?

  3. #3
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,525
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by Cbani View Post
    any quick reply guys?
    Make your thisLine1 a StringBuilder and append all your Strings to it; at the end convert your StringBuilder to a String.

    kind regards,

    Jos

  4. #4
    RamyaSivakanth's Avatar
    RamyaSivakanth is offline Senior Member
    Join Date
    Apr 2009
    Location
    Chennai
    Posts
    835
    Rep Power
    6

    Default

    Instead of building the string inside the method n number of times..u can build one time and call the method right.No need to build like this below

    fromFileLocation+ GenericConstants.FILESEPARATOR +fileName...

    you can build the string once and pass it inside this chunkFile method right?
    Ramya:cool:

  5. #5
    Cbani is offline Member
    Join Date
    Jan 2010
    Posts
    90
    Rep Power
    0

    Default

    Make your thisLine1 a StringBuilder and append all your Strings to it; at the end convert your StringBuilder to a String.

    the above solution is not working at all...as it goes out of memory

  6. #6
    Cbani is offline Member
    Join Date
    Jan 2010
    Posts
    90
    Rep Power
    0

    Default

    Ramya,
    The chunkFile method is called only once in its life cycle...so i dont think this an issue.
    The issue is thisLine1(as it is holding lets say 10,000 records in it; where the actual file is 1,00000 and interval is 10,000) is using a lot of memory which is making the process too slow.
    Do we have any API in java file system where i could read a file from one line to the other desired line; so that i could skip the thisLine1 code...

  7. #7
    RamyaSivakanth's Avatar
    RamyaSivakanth is offline Senior Member
    Join Date
    Apr 2009
    Location
    Chennai
    Posts
    835
    Rep Power
    6

    Default

    Just go thru this Java Examples - How do I read file contents to string using commons-io? whether it helps you.

    Be specific with what you want exactly.

    Iam not able to understand
    "Do we have any API in java file system where i could read a file from one line to the other desired line; so that i could skip the thisLine1 code... "
    Ramya:cool:

  8. #8
    Cbani is offline Member
    Join Date
    Jan 2010
    Posts
    90
    Rep Power
    0

    Default

    "Do we have any API in java file system where i could read a file from one line to the other desired line; so that i could skip the thisLine1 code... "

    let me explain..

    lets say i have a file of 1,000 lines. coz of some performance issues in reading the file line by line, i want some mechanism where i could read the file lets say from line 1 to line 100 then line101 to line200...and so on.I dont wanna read the 1st 1-100 lines line by line, instead i want some mechanism where i will specify the fromLineNumber to toLineNumber and the contents in between that would be read at once..

    Hope this makes some sense

  9. #9
    RamyaSivakanth's Avatar
    RamyaSivakanth is offline Senior Member
    Join Date
    Apr 2009
    Location
    Chennai
    Posts
    835
    Rep Power
    6

    Default

    Hi,
    I don't think any api in java like this.But,for reading the entire file we are having some 3rd party api like what i mentioned below.Just have a try and let me know.

    -regards
    Ramya
    Ramya:cool:

  10. #10
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,525
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by Cbani View Post
    Make your thisLine1 a StringBuilder and append all your Strings to it; at the end convert your StringBuilder to a String.

    the above solution is not working at all...as it goes out of memory
    You didn't implement it correctly then. Don't blame the suggestion, blame your incorrect implementation thereof instead.

    kind regards,

    Jos

  11. #11
    Cbani is offline Member
    Join Date
    Jan 2010
    Posts
    90
    Rep Power
    0

    Default

    You didn't implement it correctly then. Don't blame the suggestion, blame your incorrect implementation thereof instead.
    Cool down dude..
    my intention was not to blame you rather to get a solution.Now below is the implementation as suggested by you

    public static Map<Long, String> chunkFile(String fileName, String fromFileLocation, long interval, long parentId, String toFileLocation) throws Exception {

    String thisLine;
    StringBuilder thisLine1 = new StringBuilder(GenericConstants.EMPTY_STRING);

    long countRecord = 0;
    BufferedReader brReader = null;
    ChunkAndPriorityService chunkAndPriorityService = new ChunkAndPriorityServiceImpl();
    long id = 0;
    Map<Long, String> childFileMap = new HashMap<Long, String>();

    try {

    brReader = new BufferedReader(new FileReader(fromFileLocation+ GenericConstants.FILESEPARATOR +fileName));
    countRecord = 0;
    try{
    fileName = fileName.substring(0, fileName.lastIndexOf(GenericConstants.DOT));
    }catch(Exception ex){
    fileExtension = GenericConstants.EMPTY_STRING;
    ex.printStackTrace();
    }
    while ((thisLine = brReader.readLine()) != null) {
    //thisLine1 = thisLine1 + thisLine + GenericConstants.NEXT_LINE;
    thisLine1.append(thisLine1);
    thisLine1.append(thisLine);
    thisLine1.append(GenericConstants.NEXT_LINE);
    countRecord++;

    if (countRecord == interval) {
    id = chunkAndPriorityService.getChildSequence();
    saveFileDetails(id, fileName, countRecord, parentId);
    writeFile(toFileLocation, fileName, thisLine1.toString(), id);
    childFileMap.put(id, fileName + id + fileExtension);
    countRecord = 0;
    thisLine1 = new StringBuilder(GenericConstants.EMPTY_STRING);
    }
    }
    if ((countRecord != 0) && (countRecord < interval)) {
    id = chunkAndPriorityService.getChildSequence();
    saveFileDetails(id, fileName, countRecord, parentId);
    writeFile(toFileLocation, fileName, thisLine1.toString(), id);
    childFileMap.put(id, fileName + id + fileExtension);
    }

    } catch (IOException e) {
    System.err.println("Error: " + e);
    throw e;
    } catch (Exception e) {
    throw e;
    }
    return childFileMap;
    }

    private static void writeFile(String filePath, String fileName, String fileContent, long id) throws IOException {
    File newFile;
    BufferedWriter out = null;
    FileWriter fstream = null;

    try {
    newFile = new File(filePath + GenericConstants.FILESEPARATOR + fileName + id + fileExtension);
    System.out.println("Writing file "+fileName+" @"+newFile);
    newFile.createNewFile();
    fstream = new FileWriter(newFile);
    out = new BufferedWriter(fstream);
    out.write(fileContent);
    out.close();
    } catch (IOException e) {
    out.close();
    e.printStackTrace();
    throw e;
    }

    }
    Correct me if you find anything wrong in the implementation.
    This is what is the exception log i am getting

    java.lang.OutOfMemoryError: requested array is larger than heap
    at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.
    java:99)
    at java.lang.StringBuilder.append(StringBuilder.java: 131)
    at java.lang.StringBuilder.append(StringBuilder.java: 172)
    at com.****.***.utility.ChunkFileUtil.chunkFile(Chunk FileUtil.java:1
    00)
    at com.****.****.adapters.ChunkAndPriorityServiceActi vatorBean.onMess
    age(ChunkAndPriorityServiceActivatorBean.java:172)
    at weblogic.ejb.container.internal.MDListener.execute (MDListener.java:46
    6)
    at weblogic.ejb.container.internal.MDListener.transac tionalOnMessage(MDL
    istener.java:371)
    at weblogic.ejb.container.internal.MDListener.onMessa ge(MDListener.java:
    327)
    at weblogic.jms.client.JMSSession.onMessage(JMSSessio n.java:4072)
    at weblogic.jms.client.JMSSession.execute(JMSSession. java:3964)
    at weblogic.jms.client.JMSSession$UseForRunnable.run( JMSSession.java:449
    0)
    at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapte rImpl.run(SelfTunin
    gWorkManagerImpl.java:464)
    at weblogic.work.ExecuteThread.execute(ExecuteThread. java:200)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java :172)
    Last edited by Cbani; 03-10-2010 at 07:34 AM.

  12. #12
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,525
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by Cbani View Post
    Java Code:
    				thisLine1.append(thisLine1);
    Correct me if you find anything wrong in the implementation.
    This is what is the exception log i am getting

    java.lang.OutOfMemoryError: requested array is larger than heap
    at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.
    java:99)
    That's what happens when you try to append a StringBuilder to itself; don't do that but think first before you attempt to code. The StringBuilder must be used to collect (append) the new content, there is no need to append the already present content.

    kind regards,

    Jos

  13. #13
    Cbani is offline Member
    Join Date
    Jan 2010
    Posts
    90
    Rep Power
    0

Similar Threads

  1. Performance Issue
    By hiranya in forum Threads and Synchronization
    Replies: 2
    Last Post: 11-24-2008, 11:32 PM
  2. performance is degraded in 1.6
    By jagadeeshchinni in forum New To Java
    Replies: 1
    Last Post: 10-22-2008, 04:24 PM
  3. Performance issue
    By mathes_n in forum Web Frameworks
    Replies: 8
    Last Post: 09-02-2008, 05:11 AM
  4. performance problem on osx
    By coldnebo in forum Advanced Java
    Replies: 3
    Last Post: 08-01-2008, 09:39 PM
  5. Performance Plugin
    By bugger in forum Eclipse
    Replies: 0
    Last Post: 01-31-2008, 02:01 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •