Page 2 of 2 FirstFirst 12
Results 21 to 37 of 37
  1. #21
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    Hi Tolls,

    I'm afraid my fine detailed knowledge of HTTP1.1 is about as extensive as yours !

    However, going by the documentation of the API I'm wanting to make use of (Openstack Swift Cloud Storage) , it seems to imply that certain headers are not able to be used with chunking.

    Users can upload data without needing to know in advance the amount of data to be uploaded. Users can do this by specifying an HTTP header of Transfer-Encoding: chunked and not using a Content-Length header. A good use of this feature would be doing a DB dump, piping the output through gzip, then piping the data directly into OpenStack Object Storage without having to buffer the data to disk to compute the file size.


    Whilst the above sounds great in principle (i.e. about not having to write to disk etc.), the problem is I would also loose the ability to provide the API with the MD5 checksum. And since the API used the MD5 provided to ensure the integrity of the uploaded data before returning an "OK", it's quite an important feature !

  2. #22
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    RFC 2616 - 4.4 Message length

    The Content-Length header field MUST NOT be sent if these two lengths are different (i.e., if a Transfer-Encoding header field is present). If a message is received with both a Transfer-Encoding header field and a Content-Length header field, the latter MUST be ignored.


    But I guess I can live with the loss of content length for this application. All depends on how the Etag is dealt with, so I need to go research that, becasue if the API ignores the Etag in chunked, then I'm back to square one.

  3. #23
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    Oh well, I thought it was too good to be true.

    Turned on chunked. Again, small files worked beautifully (and API seemed to be using Etag properly to verify).

    Throw my 2GB file at it and ..... yep.... OOME....

    Oh well, guess I've no choice but go have a dump now !

  4. #24
    Tolls is online now Moderator
    Join Date
    Apr 2009
    Posts
    12,006
    Rep Power
    20

    Default

    Just take it easy...remember what happened to Elvis!

  5. #25
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,515
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by Tolls View Post
    Just take it easy...remember what happened to Elvis!
    Elvis has just left the building ...

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  6. #26
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    haha, thanks guys..


    Well, I'm currently staring at Eclipse MAT feeling like I've been thrown in at the deep end. But from what I can tell thus far is that byte[] is the main culprit with 263,675,704 shallow heap and >= 263,675,704 retained heap, 99.04% percentage.

    Time to find a good MAT tutorial me thinks.

  7. #27
    Tolls is online now Moderator
    Join Date
    Apr 2009
    Posts
    12,006
    Rep Power
    20

    Default

    MAT will allow you to drill up from that byte[] to who owns it...and that'll be the output stream I'd bet.
    Which sort of doesn't help if you can't flush the stream with chunked data.

  8. #28
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    to who owns it...and that'll be the output stream I'd bet.
    Pretty much....

    Java Code:
    byte[]
    java.lang.Thread
    sun.net.www.http.PosterOutputStream
    java.lang.Thread
    which kind of correlates with the stdout stacktrace.....

    Java Code:
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
            at java.util.Arrays.copyOf(Unknown Source)
            at java.io.ByteArrayOutputStream.write(Unknown Source)
            at sun.net.www.http.PosterOutputStream.write(Unknown Source)

    and this is my present coding attempt....

    Java Code:
                       this.ssl.conn.setRequestProperty("Transfer-Encoding", "chunked" );
                        this.ssl.conn.connect();
                        this.inStream = new FileInputStream(this.fil);
                        this.outStream = this.ssl.conn.getOutputStream();
                        int r = 0;
                        byte[] barray = new byte[8024];
                        while ((r = this.inStream.read(this.barray)) >= 0) {
                            this.outStream.write(this.barray, 0, r);
                            this.outStream.flush();
                        }
    I can't flush any more frequently than that !

  9. #29
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,515
    Blog Entries
    7
    Rep Power
    20

    Default

    @OP: did you write the server part yourself? i.e. can you add a bit of functionality to it?

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  10. #30
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    Hi JosAH,

    Unfortunatley not, and although it's an open source server part, it's written in Python something which I don't even know how to write a hello world in !

    I'm currently working trying out Apache HTTPClient as a possible solution

  11. #31
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,515
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by HeapSpace View Post
    Unfortunatley not, and although it's an open source server part, it's written in Python something which I don't even know how to write a hello world in !
    Too bad because I had the following (silly) idea: send the big files in parts file00, file01, file02 etc. and add a small command to the server, e.g. join file00 file01 file02 etc. that joins the file parts on the server side again. Maybe you can find a pythonian that can do the file joining part for you ;-)

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  12. #32
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    Interesting idea Jos, although I think I'll keep server code hacking as a last resort.

    I've currently got a 2GB non-chunked upload going on via my reworked Apache HTTPClient based code...... so far, so good...... although knowing my luck, as soon as I post this message my JVM will exit with an OOM !

  13. #33
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default



    Apache HTTPClient rules !!!!!!!

    2GB ... no sweat with Apache..... I think URLConnection will be consigned to the trash in my code !

  14. #34
    Tolls is online now Moderator
    Join Date
    Apr 2009
    Posts
    12,006
    Rep Power
    20

    Default

    Sounds a reasonable decision there.
    And you didn't go the way of Elvis, which has got to be seen as a bonus.

  15. #35
    HeapSpace is offline Member
    Join Date
    Jun 2011
    Posts
    20
    Rep Power
    0

    Default

    Thanks for all your help.

    And indeed, I didn't go the way of Elvis, so I live to fight another day (and come back here to haunt you like a ghost with more Java questions ... mwahahahahaa ..... )

  16. #36
    Tolls is online now Moderator
    Join Date
    Apr 2009
    Posts
    12,006
    Rep Power
    20

    Default

    At least yours are more interesting than the usual "What does ClassNotFoundException mean"?

  17. #37
    JosAH's Avatar
    JosAH is online now Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,515
    Blog Entries
    7
    Rep Power
    20

    Default

    Quote Originally Posted by Tolls View Post
    At least yours are more interesting than the usual "What does ClassNotFoundException mean"?
    You forgot "javac is not recognized as an internal or external etc." or "how can I make an .exe file?" ...

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

Page 2 of 2 FirstFirst 12

Similar Threads

  1. Replies: 0
    Last Post: 02-08-2011, 08:51 AM
  2. how to split large xml file into small xml file in java
    By enggvijaysingh@gmail.com in forum XML
    Replies: 2
    Last Post: 02-07-2011, 09:34 AM
  3. post of large xml file on third party webservices
    By enggvijaysingh@gmail.com in forum XML
    Replies: 6
    Last Post: 11-16-2010, 03:03 PM
  4. Replies: 0
    Last Post: 04-25-2009, 10:18 PM
  5. Replies: 1
    Last Post: 08-07-2007, 05:37 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •