Writing huge sized file data(more than 100Mb) to a output stream(converting to byte[]

[Chinnu55]

Hi All,

I have a file which is more than 100MB which needs to be written to a output stream (ServletOutputStream) as a response to a request where we can write byte array to a output stream. (outputStream.write(byte[])

Currently, i am using apache commons IOUtils package,

IOUtils.toByteArray(InputStream inputStream)
but this is not working when the file size is more.


Please let me know the better approach with good performance to handle huge files like more than 100MB.

Thanks,
Chinnu

[Eranga]

Have a look at the following.

http://code.hammerpig.com/how-to-rea...s-in-java.html

It's a very common solution, yet it has some drawbacks as well.

[Eranga]

Have a look at the following.

How to Read Really Large Files in Java | Code Comments

It's a very common solution, yet it has some drawbacks as well.

[KevinWorkman]

Crossposted: Writing huge sized file data(more than 100Mb) to a output stream(converting to byte[]

[Eranga]

Oh no. Wasting my time.

[Tolls]

Where is this file coming from, and do you need to read it all in in one go before streaming it out?
If it's simply passing through, then read and write at the same time.

[Chinnu55]

Kevin - Thanks for making me understand about Crosspost.

I am currently using java 1.4.

I have huge files whose size is more than > 150 MB. I need to write it to output response so that user can download the report in excel format.
But, to write to response outputstream

OutputStream out
out.write(byte[]) - outputStream accepts byte[].

So, i need to convert huge file data into byte[] first to pass it to response.

Tolls, i am generating file locally with the data.

Thanks,
Chinnu

[Tolls]

Wrap the OutputStream in an OutputStreamWriter (or something along those lines) and use that.

Then write to that stream as you generate the data. Don't (if at all possible, and it almost always is) wait until you have all the data and then write. That will mean you can handle any size of data.

[Chinnu55]

Thanks Tolls for your reply,

Problem is :

I have a file which will be more than 100 MB based on data coming from DB, i need to put this file in a zip file.


while ((newEntry = inZip.getNextEntry()) != null)
{
destEntry = new ZipEntry (newEntry.getName());
ZipEntry out;
InputStream source;
if (destEntry.getName().equals("HugeFile.xml"))
{
out = new ZipEntry(destEntry);
out.setSize(HugeFilesByteData.length);
// HugeFilesByteArrayData is a HugeFile's data converted to byte array
source = new ByteArrayInputStream(HugeFilesByteArrayData);

}
else
{
/* just copy from source ZIP */
out = destEntry;
source = inZip;
}
outZip.putNextEntry(out);
IOUtils.copy(source,outZip);

To put Huge file into Zip file, i need to convert the file data into byte[].

byte[] fileBytes = null;
try {
fileBytes = new byte[(int) HugeFile.length()]; // here it fails and throws outofmemory error.
fileInStream = new FileInputStream(inputFile);
fileInStream.read(fileBytes);
}

and then this Zip file which contains many files along with HugeFile.xml will be sent as a response through outputstream.

please let me know the better approach.

Thanks,
Chinnu

[Tolls]

Don't read it all in one go.
This is what I have been telling you.

Open input stream
Open output stream
create byte buffer
while (read stuff into byte buffer) {
    write byte buffer to output stream
}

That's how you do this.
You use a buffer (say 8k).
Read into that, write it out.
Repeat until you run out of things to read in.

[Chinnu55]

May i know what will be the maximum size byte[] can take?

byte b[] = new byte[???];

Thanks,
Chinnu

[Tolls]

Why?
Just use an 8k buffer. It's good enough for almost any situation.

byte[] b = new byte[8192];

[j2me64]

there are classes that let you define filters for compressing data to zip, examples here.

[Chinnu55]

Thanks Tolls,

I am still getting confused how to move forward.

Can you please give a sample code which might help?

Thanks,
Chinnu

[Tolls]

I have explained above how to write it.
So try.

[Chinnu55]

Thanks Tolls.