data loss in the secure file transfer using java

[antony75]

Hi,

My java Appllication uses the sftp(secure file transfer protocol) for file transferring. when the load is high like 150 GB , data loss occuring for some files especially last chunk of data is missing.

Core file transfer code

while (true)

{

int read = bis.read(buffer); //Reading bytes into the byte array (buffer) from network stream (bis).

if (read==-1) {

break; //Break the loop, if there are no more bytes available in the InputStream.

}

bos.write(buffer,0,read); //Writing the same number of bytes what we read into buffer byte array into OutputStream

complete.update(buffer,0,read); //Giving the same number of bytes what we read above to the checksum calculation.

bytesSoFar += read; //Incrementing the bytes that we have read from the source.


//if the bytes transferred are equal to the source file size

If (bytesSoFar == fileInfo.getFileSize().intValue())

{

// Calculate the file checksum.

// Update DbAuditLog with checksum and no. of bytes read

}

else

{

// Update DbAuditLog with no. of bytes read

}

}

bos.flush();


Please help , why the data loss is happening when the big size files transfer

Note:

1.150 gb not a single file multiple files.

2.SFTP no java specific API, So connection is happening through java .io.sockect connection

Due to data loss, checksum also not correct.
This issue not occurring continuously.. some times whole transfer failing or success.

Is NFS mount plays any role when this occurrence happening .. OS is HP-UX ia64 itanium version.

Now we are observing with glance tool which is provided by hp to analyze system activity

Application is running along with jconsole.

All the transaction are running under multithreaded way.. that means,most of the txns run multi threaded way , few txns run under singel thread. it depends on the threads available in the thread pool.

[fishtoprecords]

You need to make sure you are reading any partially filled last buffer blocks.

SFTP is just SSH + FTP. What is doing your encryption?

How big are these files? Sending things bigger than 2GB can cause all sorts of issues.

[jfileupload]

You should make a test with an applet that supports SFTP such as JFileUpload (there is a free full featured trial). If it works then it means that you have a problem in your code.

[CJSLMAN]

Don't know if it will help with your problem, but you should always close you outputstreams:

bos.close();

You should also close your program's inputstream when finished.

Luck,
CJSL

[toadaly]

My guess is an exception is being thrown that you are not handling. If the socket disconnects mid transfer (which can happen), or if you get a file write error (which can also happen), the code you have here will result in a partial file.