Results 1 to 16 of 16
  1. #1
    Chinnu55 is offline Member
    Join Date
    Apr 2011
    Posts
    20
    Rep Power
    0

    Default Writing huge sized file data(more than 100Mb) to a output stream(converting to byte[]

    Hi All,

    I have a file which is more than 100MB which needs to be written to a output stream (ServletOutputStream) as a response to a request where we can write byte array to a output stream. (outputStream.write(byte[])

    Currently, i am using apache commons IOUtils package,

    IOUtils.toByteArray(InputStream inputStream)
    but this is not working when the file size is more.


    Please let me know the better approach with good performance to handle huge files like more than 100MB.

    Thanks,
    Chinnu

  2. #2
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

  3. #3
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

  4. #4
    KevinWorkman's Avatar
    KevinWorkman is offline Crazy Cat Lady
    Join Date
    Oct 2010
    Location
    Washington, DC
    Posts
    3,864
    Rep Power
    8

    Default

    How to Ask Questions the Smart Way
    Static Void Games - Play indie games, learn from game tutorials and source code, upload your own games!

  5. #5
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

  6. #6
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,792
    Rep Power
    19

    Default

    Where is this file coming from, and do you need to read it all in in one go before streaming it out?
    If it's simply passing through, then read and write at the same time.

  7. #7
    Chinnu55 is offline Member
    Join Date
    Apr 2011
    Posts
    20
    Rep Power
    0

    Default

    Kevin - Thanks for making me understand about Crosspost.

    I am currently using java 1.4.

    I have huge files whose size is more than > 150 MB. I need to write it to output response so that user can download the report in excel format.
    But, to write to response outputstream

    OutputStream out
    out.write(byte[]) - outputStream accepts byte[].

    So, i need to convert huge file data into byte[] first to pass it to response.

    Tolls, i am generating file locally with the data.

    Thanks,
    Chinnu

  8. #8
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,792
    Rep Power
    19

    Default

    Wrap the OutputStream in an OutputStreamWriter (or something along those lines) and use that.

    Then write to that stream as you generate the data. Don't (if at all possible, and it almost always is) wait until you have all the data and then write. That will mean you can handle any size of data.

  9. #9
    Chinnu55 is offline Member
    Join Date
    Apr 2011
    Posts
    20
    Rep Power
    0

    Default Writing huge sized file data(more than 100Mb) to a zip file(converting to byte[]

    Thanks Tolls for your reply,

    Problem is :

    I have a file which will be more than 100 MB based on data coming from DB, i need to put this file in a zip file.


    while ((newEntry = inZip.getNextEntry()) != null)
    {
    destEntry = new ZipEntry (newEntry.getName());
    ZipEntry out;
    InputStream source;
    if (destEntry.getName().equals("HugeFile.xml"))
    {
    out = new ZipEntry(destEntry);
    out.setSize(HugeFilesByteData.length);
    // HugeFilesByteArrayData is a HugeFile's data converted to byte array
    source = new ByteArrayInputStream(HugeFilesByteArrayData);

    }
    else
    {
    /* just copy from source ZIP */
    out = destEntry;
    source = inZip;
    }
    outZip.putNextEntry(out);
    IOUtils.copy(source,outZip);

    To put Huge file into Zip file, i need to convert the file data into byte[].

    byte[] fileBytes = null;
    try {
    fileBytes = new byte[(int) HugeFile.length()]; // here it fails and throws outofmemory error.
    fileInStream = new FileInputStream(inputFile);
    fileInStream.read(fileBytes);
    }

    and then this Zip file which contains many files along with HugeFile.xml will be sent as a response through outputstream.

    please let me know the better approach.

    Thanks,
    Chinnu

  10. #10
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,792
    Rep Power
    19

    Default

    Don't read it all in one go.
    This is what I have been telling you.

    Java Code:
    Open input stream
    Open output stream
    create byte buffer
    while (read stuff into byte buffer) {
        write byte buffer to output stream
    }
    That's how you do this.
    You use a buffer (say 8k).
    Read into that, write it out.
    Repeat until you run out of things to read in.

  11. #11
    Chinnu55 is offline Member
    Join Date
    Apr 2011
    Posts
    20
    Rep Power
    0

    Default Writing huge sized file data(more than 100Mb) to a zip file(converting to byte[]

    May i know what will be the maximum size byte[] can take?

    byte b[] = new byte[???];

    Thanks,
    Chinnu

  12. #12
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,792
    Rep Power
    19

    Default

    Why?
    Just use an 8k buffer. It's good enough for almost any situation.
    Java Code:
    byte[] b = new byte[8192];

  13. #13
    j2me64's Avatar
    j2me64 is offline Senior Member
    Join Date
    Sep 2009
    Location
    Zurich, Switzerland
    Posts
    962
    Rep Power
    5

    Default

    there are classes that let you define filters for compressing data to zip, examples here.

  14. #14
    Chinnu55 is offline Member
    Join Date
    Apr 2011
    Posts
    20
    Rep Power
    0

    Default Writing huge sized file data(more than 100Mb) to a zip file(converting to byte[]

    Thanks Tolls,

    I am still getting confused how to move forward.

    Can you please give a sample code which might help?

    Thanks,
    Chinnu

  15. #15
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,792
    Rep Power
    19

    Default

    I have explained above how to write it.
    So try.

  16. #16
    Chinnu55 is offline Member
    Join Date
    Apr 2011
    Posts
    20
    Rep Power
    0

Similar Threads

  1. Writing output file
    By doymand in forum New To Java
    Replies: 2
    Last Post: 11-05-2010, 01:24 PM
  2. converting byte array to bmp file
    By Moorkh in forum New To Java
    Replies: 2
    Last Post: 09-07-2010, 02:58 PM
  3. [SOLVED] Converting output stream to string
    By dan0 in forum New To Java
    Replies: 0
    Last Post: 03-16-2009, 03:44 PM
  4. Replies: 3
    Last Post: 02-16-2009, 09:20 PM
  5. Reading/Writing a File using byte array
    By Java Tip in forum Java Tip
    Replies: 0
    Last Post: 01-16-2008, 10:41 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •