Axis 1.4 Larg Attachments out of Memory

[BSA]

I have been struggling with large attachment processing for some time using Axis 1.4. My client is trying to send up to 500 megabyte files to my webservice and I have overcome most hurdles to get this accomplished including client and server side timeout modifications, memory allocation, disk space accommodations, etc... I have finally found a way to send these attachments, but it will only work when I send them while in Netbeans 6.7.1.

To make it quick, when I include the axis-ant.jar file to my classpath the memory usage remains stable and very low, about ~10 megabytes, and the attachment is successfull. When I leave this file out of my classpath everything compiles and runs, but at the time my client actually makes the call to the webservice (datahandler already setup) the memory usage jumps from ~10 megabytes to 1 gigabyte in a few short seconds.

The inclusion of the axis-ant.jar file, and why this works, is a bit of a mystery to me. Furthermore, I wonder why it only works when running in the Netbeans IDE. When I run this from the command line and actually use the -cp parameter to specify my classpath including all the appropriate files it will always throw a Java heap space OutOfMemoryError error.

Has anyone encountered this and is there any suggested solution?

I do not have an option of Axis 2 or Spring-WS on this project and must remain on Axis 1.4 for now.

NOTE: Small files up to 100 megabytes have always been successfull.

[travishein]

I don't think axis 1.4 supports the message attachments does it ? in that when you put the contents inside the SOAP envelope, the contents must be parsed through the (yuck,) DOM parser. Yes, axis 2 has the StAX parser and does support attachments after the XML SOAP envelope, which allows the larger content to be sent, but that also currently breaks when going through ws-secure plugin.. and its moot because you can't upgrade to this now anyway.

One time I invented a kind of support servlet that was my own URL endpoint for file transfer. the short version is the remote client would request a file to download, the SOAP service would reply with the URL to my download servlet, including any URL parameters that were needed. at the same time, the server-side logic would create a kind of 'request token' that would create a one time temporary credentials allow the client session to invoke my custom file download servlet in the near future. The client would then receive the SOAP reply, unpack the URL and use its own HTTP(s) client to connect to, authenticate on the custom file download servlet, and then the file gets sent.
The similar logic for uploading files, where teh clinet posts a SOAP request, the server acknowledges with a response containing the URL and any parameters required for the cliient to soon connect to. the client unpacks the response and invokes a HTTP POST to this upload servlet.

The out of band approach gets past all the nasty stickiness with large files and axis 1.x. Though it does require the additional custom things on both the server and all the clients. Its bascally you are inventing your own custom protocol exension that just happens to use SOAP web services. and it is really only valid if all of the clients are able to implement something to conform to your adhoc methods. IN my case it was suitable since I was building the only client to consume the service and I had free reign over how things went together. in the end it was more elegant than I had first thought, and certianly better than using some kind of sibling FTP file transfer method. I mean, everything still worked over the single web server port anyway. after a while I realized what a pig the entire SOAP stack was and now I just use a JSON-RPC facility instead with the custom file servlets, much faster too. but I digress.

[Tolls]

There's something similar here, for image downloads. It uses a timeout on the url (well, it removes the image file after a couple of minutes anyway).

SOAP's great, but it was never intended for large files. That's an abuse of it's original design, frankly.

[BSA]

Thanks for the suggestion(s)/response. I wish I could change API protocol and or methods in the webservices for this project... I can't at this point as the cascading implications are too great.

If I can figure out how my IDE manages to do it succesfully then I should be good. I just seems that within the IDE the DataHandler actually streams the data from disk to the endpoint as I want it to, but outside the IDE it wants to load the entire file into memory, prepare it to be sent, and them stream it... this of course need an extremely large amount of memory to accomplish.

Does an IDE like netbeans provide additional memory managment when running your application from withing the IDE? When I do a (-verbose) execution within netbeans there are a number of additional classes that report being loaded like an ObjectDataContentHandler or ObjectStreamException... and more.

[travishein]

i wonder, it might have a different XML parser .jar file, like one that has a different implementation for the org.w3c.dom.* or org.apache.xerces.parsers, or what not. Or maybe it is setting a system property that enables some file based caching ? I'm stumped there.

So, you can't changed existing API, but you also can't add new API, like a new version_2 endpoint url, like only to have the getLargeFile() funciton for now and to start to transition into this one as you have the time ?

[BSA]

This is getting more curious as I test. It may be an Axis memory cleanup issue on the client side.

I noticed today that I actually sent a 500MB file to the service. All 500MB was received at the server side and stored in Axis' temporary file system directory. When the web service code, my server side code, actually started processing it the client side then received an Out Of Memory error. I am suspecting that Axis told my client that it had received all the data an that it was safe to cleanup any memory on the client side... then Axis blew the heap on the client.

I actually have other developers here that can do this without an error. The only difference is that they are running their applications within a web server which I am suspecting helps manage their memory just like my Netbeans did when successfully test. In production, again, java is called straight from the command line.

The web services actually continued to process the complete file and it is now in my database...

[Tolls]

But Axis can't "blow the heap". It doesn't control memory allocation any more than any code you write controls memory allocation. It can't say "OK to cleanup any memory" because you can't.

[mtanzania]

BSA I am curious as to how you ended up solving your issue I am facing a similar problem and with axis 1.4 it doesn't seem like I can send anything above 1MB for me. Would you mind sharing your client code that sends these attachments to the server? Thanks