Results 1 to 10 of 10
  1. #1
    juamd is offline Member
    Join Date
    Apr 2010
    Posts
    25
    Rep Power
    0

    Default Load file with a lot or lines.

    Hi everybody

    I need to load a file which contains 1000000 lines. I always load files using a string variable to store files but this file is larger than any other I have loaded before and I need to know the best way to do that, any suggestions ?


    Greetings !!!!

  2. #2
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,368
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by juamd View Post
    I need to load a file which contains 1000000 lines. I always load files using a string variable to store files but this file is larger than any other I have loaded before and I need to know the best way to do that, any suggestions ?
    If you can process your file line by line there's no need to load the file contents at all:

    Java Code:
    BufferedReader br= ...;
    String line;
    while ((line= br.readLine()) != null)
       process(line);
    Otherwise you have to tell us some more ...

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  3. #3
    juamd is offline Member
    Join Date
    Apr 2010
    Posts
    25
    Rep Power
    0

    Default

    Thanks Jos.. so I am wondering if there is a way to do that much faster, read the file line by line will take a lot of time and the process needs to be as fast as possible.

  4. #4
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,368
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by juamd View Post
    Thanks Jos.. so I am wondering if there is a way to do that much faster, read the file line by line will take a lot of time and the process needs to be as fast as possible.
    Well, reading line by line using a BufferedReader isn't that slow because that Reader reads 4K or 8K chunks per actual read. There must be something else that's slowing the process down ...

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  5. #5
    juamd is offline Member
    Join Date
    Apr 2010
    Posts
    25
    Rep Power
    0

    Default

    I'll review the problem again.

    Thanks friend.

  6. #6
    Jeeni101 is offline Member
    Join Date
    Apr 2011
    Posts
    22
    Rep Power
    0

    Default

    Try:


    int size = (int)f.length();
    byte[] bytes = new byte[size];
    InputStream in = new BufferedInputStream(new FileInputStream(f));
    in.read(bytes);


    Be aware that the maximum number of bytes in the file can be no more than Integer.MAX_VALUE, and you may have to up the memory of your app.

    Or if you want super efficency:
    FileChannel fc = new FileInputStream(f).getChannel();
    ByteBuffer buffer = ByteBuffer.allocate(size);
    fc.read(buffer);

    Both these methods will read the whole file into memory and then you can mess around with it to your hearts content.
    Last edited by Jeeni101; 04-25-2011 at 04:17 PM.

  7. #7
    juamd is offline Member
    Join Date
    Apr 2010
    Posts
    25
    Rep Power
    0

    Default

    Thanks a lot.

  8. #8
    juamd is offline Member
    Join Date
    Apr 2010
    Posts
    25
    Rep Power
    0

    Default Load file with a lot of lines.

    I loaded the file using getChannel it looks like this:

    Java Code:
                FileInputStream f = new FileInputStream(this.txtArchivo.getText());
                FileChannel inChannel = f.getChannel();
                int bytesRead = 200;
                ByteBuffer buffer = ByteBuffer.allocate(200);
    
                while (bytesRead != -1) {
                    buffer.flip();
    
                    while (buffer.hasRemaining()) {
                        char c = (char) buffer.get();
                        int valor = (int) c;
    
                        System.out.print(c);
                    }
                    buffer.clear();
                    bytesRead = inChannel.read(buffer);
                }
                f.close();
    It let me load the file and then in the while loop show the content of it, but I need to take the file line by line, so I tried to add a clause if like this:

    Java Code:
    //valor stores ascii code of caracter read
    if (valor != 13){
       //add the char to a string builder object
    }
    But it did not work.

    Any suggestions ?


    Greetings!!

  9. #9
    Jeeni101 is offline Member
    Join Date
    Apr 2011
    Posts
    22
    Rep Power
    0

    Default Hope this helps

    Try this:
    The biggest differance is that I used '\n' as the CR charactor rather than testing for the ascii value. '\n' should be more reliable.
    Other things to note are the sizing of the buffer.

    I'm not sure where you are trying to get to, but you could conver the buffer to a String and use the replace method to replace CRs with white space. Thats the commented out part.



    File file = new File("C:/Temp/big/BigFile.txt");
    long size = file.length();
    if(size > Integer.MAX_VALUE) {
    System.out.println("File to big");
    System.exit(0);
    }
    final int maxBufferSize = 1048576;
    long bufferSize = size > maxBufferSize ? maxBufferSize : size;

    FileInputStream f = new FileInputStream(file.getAbsolutePath());
    FileChannel inChannel = f.getChannel();
    int bytesRead = 0; // No bytes read yet!!

    // Note that bufferSize can only be a maximum of megaByte as par the inialisation above.
    ByteBuffer buffer = ByteBuffer.allocate((int)bufferSize);

    int crCount = 0; // Just a variable to count the carriage return characters.
    bytesRead = inChannel.read(buffer);// To prime things before going into the loop
    while (bytesRead != -1) {
    buffer.flip();

    // int bytesInBuffer = buffer.limit();
    // byte[] bufferBytes = new byte[bytesInBuffer];
    // buffer.get(bufferBytes);
    // String bufferContentsWithNoCarriageReturns = new String(bufferBytes).replace('\n', ' ');
    // System.out.println(bufferContentsWithNoCarriageRet urns);

    while (buffer.hasRemaining()) {
    char valor = (char) buffer.get();

    if (valor != '\n'){
    //add the char to a string builder object
    } else {
    crCount++;
    }
    }
    buffer.clear();
    bytesRead = inChannel.read(buffer);
    }
    f.close();
    System.out.println("Done. Carriage Returns Found: " + crCount);


    Oh, and by the way, the slowness you may be finding could be down to the processing of the Strings when you get your hands on the lines in the file.
    Last edited by Jeeni101; 04-26-2011 at 07:00 PM.

  10. #10
    juamd is offline Member
    Join Date
    Apr 2010
    Posts
    25
    Rep Power
    0

    Default Load file with a lot of lines.

    Hi !!!

    I solved the problem doing this:

    Java Code:
    if (valor != 13){
        //add to StringBuilder object
    }else{
        //set stringBuilder's length to 0 and send the data to the database.
    }
    Thanks a lot for too much help.

Similar Threads

  1. Two '\n' characters between lines while reading File
    By subith86 in forum New To Java
    Replies: 6
    Last Post: 02-26-2011, 09:56 AM
  2. how to remove duplicate lines from a file
    By trkece in forum New To Java
    Replies: 11
    Last Post: 02-16-2011, 06:01 AM
  3. How to remove 2 last lines in a text file?
    By Marius in forum New To Java
    Replies: 2
    Last Post: 11-30-2008, 03:54 PM
  4. Remove duplicate lines from a text file
    By Dirt.Diver in forum New To Java
    Replies: 15
    Last Post: 06-25-2008, 02:08 PM
  5. Replies: 0
    Last Post: 04-06-2008, 07:45 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •