Load file with a lot or lines.

[juamd]

Hi everybody

I need to load a file which contains 1000000 lines. I always load files using a string variable to store files but this file is larger than any other I have loaded before and I need to know the best way to do that, any suggestions ?


Greetings !!!!

[JosAH]

I need to load a file which contains 1000000 lines. I always load files using a string variable to store files but this file is larger than any other I have loaded before and I need to know the best way to do that, any suggestions ?

If you can process your file line by line there's no need to load the file contents at all:

BufferedReader br= ...;
String line;
while ((line= br.readLine()) != null)
   process(line);

Otherwise you have to tell us some more ...

kind regards,

Jos

[juamd]

Thanks Jos.. so I am wondering if there is a way to do that much faster, read the file line by line will take a lot of time and the process needs to be as fast as possible.

[JosAH]

Thanks Jos.. so I am wondering if there is a way to do that much faster, read the file line by line will take a lot of time and the process needs to be as fast as possible.

Well, reading line by line using a BufferedReader isn't that slow because that Reader reads 4K or 8K chunks per actual read. There must be something else that's slowing the process down ...

kind regards,

Jos

[juamd]

I'll review the problem again.

Thanks friend.

[Jeeni101]

Try:


int size = (int)f.length();
byte[] bytes = new byte[size];
InputStream in = new BufferedInputStream(new FileInputStream(f));
in.read(bytes);

Be aware that the maximum number of bytes in the file can be no more than Integer.MAX_VALUE, and you may have to up the memory of your app.

Or if you want super efficency:
FileChannel fc = new FileInputStream(f).getChannel();
ByteBuffer buffer = ByteBuffer.allocate(size);
fc.read(buffer);

Both these methods will read the whole file into memory and then you can mess around with it to your hearts content.

[juamd]

Thanks a lot.

[juamd]

I loaded the file using getChannel it looks like this:

            FileInputStream f = new FileInputStream(this.txtArchivo.getText());
            FileChannel inChannel = f.getChannel();
            int bytesRead = 200;
            ByteBuffer buffer = ByteBuffer.allocate(200);

            while (bytesRead != -1) {
                buffer.flip();

                while (buffer.hasRemaining()) {
                    char c = (char) buffer.get();
                    int valor = (int) c;

                    System.out.print(c);
                }
                buffer.clear();
                bytesRead = inChannel.read(buffer);
            }
            f.close();

It let me load the file and then in the while loop show the content of it, but I need to take the file line by line, so I tried to add a clause if like this:

//valor stores ascii code of caracter read
if (valor != 13){
   //add the char to a string builder object
}

But it did not work.

Any suggestions ?


Greetings!!

[Jeeni101]

Try this:
The biggest differance is that I used '\n' as the CR charactor rather than testing for the ascii value. '\n' should be more reliable.
Other things to note are the sizing of the buffer.

I'm not sure where you are trying to get to, but you could conver the buffer to a String and use the replace method to replace CRs with white space. Thats the commented out part.



File file = new File("C:/Temp/big/BigFile.txt");
long size = file.length();
if(size > Integer.MAX_VALUE) {
System.out.println("File to big");
System.exit(0);
}
final int maxBufferSize = 1048576;
long bufferSize = size > maxBufferSize ? maxBufferSize : size;

FileInputStream f = new FileInputStream(file.getAbsolutePath());
FileChannel inChannel = f.getChannel();
int bytesRead = 0; // No bytes read yet!!

// Note that bufferSize can only be a maximum of megaByte as par the inialisation above.
ByteBuffer buffer = ByteBuffer.allocate((int)bufferSize);

int crCount = 0; // Just a variable to count the carriage return characters.
bytesRead = inChannel.read(buffer);// To prime things before going into the loop
while (bytesRead != -1) {
buffer.flip();

// int bytesInBuffer = buffer.limit();
// byte[] bufferBytes = new byte[bytesInBuffer];
// buffer.get(bufferBytes);
// String bufferContentsWithNoCarriageReturns = new String(bufferBytes).replace('\n', ' ');
// System.out.println(bufferContentsWithNoCarriageRet urns);

while (buffer.hasRemaining()) {
char valor = (char) buffer.get();

if (valor != '\n'){
//add the char to a string builder object
} else {
crCount++;
}
}
buffer.clear();
bytesRead = inChannel.read(buffer);
}
f.close();
System.out.println("Done. Carriage Returns Found: " + crCount);

Oh, and by the way, the slowness you may be finding could be down to the processing of the Strings when you get your hands on the lines in the file.

[juamd]

Hi !!!

I solved the problem doing this:

if (valor != 13){
    //add to StringBuilder object
}else{
    //set stringBuilder's length to 0 and send the data to the database.
}

Thanks a lot for too much help.