Results 1 to 3 of 3
  1. #1
    rsantana is offline Member
    Join Date
    Aug 2011
    Rep Power

    Question Best practices in handling big data streams

    Hello everybody.

    I need to develop a mechanism to receive the content of a big file through the input stream of a Servlet, maybe transform the data, and then sned this to an ending stream (like a file, another URL, ...).

    I know how to read from the servlet stream. But suppose you have to handle thousands of megabytes of data. What would be the best method to handle this data? I can't load all the data, because memory will run out. I think I could store data into a temporary file, and then handle it later, but would it be a good practice? Remember that i'm working with servlets, so there is a Tomcat present, and in my local workspace I do what I want in my filesystem. Not the same about customer's workspace (I'm very limited about filesystem privileges).


  2. #2
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Rep Power


    Depends what you do with the data.
    Is it per-line processing?
    Do you have to be able to go back and forth through the data?

    That defines what you do.

  3. #3
    Hibernate's Avatar
    Hibernate is offline Senior Member
    Join Date
    Dec 2010
    Stockholm, Sweden
    Blog Entries
    Rep Power


    RandomAccessFile is perfect for jumping into random* places in a file, and it can handle large file; it is not restricted to integer, but to long, in contrast to streams.
    Remember to not read or write data byte by byte (via files or sockets), it is extremely slow, do it groups about 1 << 16 (2¹⁶) bytes (not that I'm not sure this is optimal for sockets, nor for files, you can test what the optimal size is); for files, you may consider asking to file system what the file's optimal I/O block size is. (I do not know have to do that in pure Java).

    Edit: Also, buffered streams may be usable, I do not know, since, I never actually used them, in all my years…

    *random: not sequential
    Last edited by Hibernate; 08-11-2011 at 03:36 PM.
    Ex animo! Hibernate
    Java, Arch Linux, C, GPL v3, Bash, Eclipse, Linux VT, GNOME 2 and many buttons on windows.

Similar Threads

  1. File handling-streams
    By Dayanand in forum New To Java
    Replies: 7
    Last Post: 03-25-2011, 09:56 AM
  2. Best Practices of handling more number of if..elseif.. Condition
    By gavaskarrathnam in forum Advanced Java
    Replies: 16
    Last Post: 01-28-2011, 11:25 AM
  3. Socket, streams and data types?
    By Tomtefaen in forum Networking
    Replies: 0
    Last Post: 01-15-2011, 12:36 AM
  4. Runtime.exec(), handling input and output streams
    By crookshank in forum New To Java
    Replies: 0
    Last Post: 06-05-2008, 02:41 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts