Results 1 to 5 of 5
  1. #1
    antony75 is offline Member
    Join Date
    Oct 2008
    Posts
    1
    Rep Power
    0

    Default data loss in the secure file transfer using java

    Hi,

    My java Appllication uses the sftp(secure file transfer protocol) for file transferring. when the load is high like 150 GB , data loss occuring for some files especially last chunk of data is missing.

    Core file transfer code

    while (true)

    {

    int read = bis.read(buffer); //Reading bytes into the byte array (buffer) from network stream (bis).

    if (read==-1) {

    break; //Break the loop, if there are no more bytes available in the InputStream.

    }

    bos.write(buffer,0,read); //Writing the same number of bytes what we read into buffer byte array into OutputStream

    complete.update(buffer,0,read); //Giving the same number of bytes what we read above to the checksum calculation.

    bytesSoFar += read; //Incrementing the bytes that we have read from the source.


    //if the bytes transferred are equal to the source file size

    If (bytesSoFar == fileInfo.getFileSize().intValue())

    {

    // Calculate the file checksum.

    // Update DbAuditLog with checksum and no. of bytes read

    }

    else

    {

    // Update DbAuditLog with no. of bytes read

    }

    }

    bos.flush();


    Please help , why the data loss is happening when the big size files transfer

    Note:

    1.150 gb not a single file multiple files.

    2.SFTP no java specific API, So connection is happening through java .io.sockect connection

    Due to data loss, checksum also not correct.
    This issue not occurring continuously.. some times whole transfer failing or success.

    Is NFS mount plays any role when this occurrence happening .. OS is HP-UX ia64 itanium version.

    Now we are observing with glance tool which is provided by hp to analyze system activity

    Application is running along with jconsole.

    All the transaction are running under multithreaded way.. that means,most of the txns run multi threaded way , few txns run under singel thread. it depends on the threads available in the thread pool.

  2. #2
    fishtoprecords's Avatar
    fishtoprecords is offline Senior Member
    Join Date
    Jun 2008
    Posts
    571
    Rep Power
    7

    Default

    You need to make sure you are reading any partially filled last buffer blocks.

    SFTP is just SSH + FTP. What is doing your encryption?

    How big are these files? Sending things bigger than 2GB can cause all sorts of issues.

  3. #3
    jfileupload is offline Member
    Join Date
    Mar 2008
    Posts
    10
    Rep Power
    0

    Lightbulb Try with a working Java SFTP tool

    You should make a test with an applet that supports SFTP such as JFileUpload (there is a free full featured trial). If it works then it means that you have a problem in your code.

  4. #4
    CJSLMAN's Avatar
    CJSLMAN is offline Moderator
    Join Date
    Oct 2008
    Location
    Mexico
    Posts
    1,159
    Rep Power
    7

    Default

    Don't know if it will help with your problem, but you should always close you outputstreams:
    Java Code:
    bos.close();
    You should also close your program's inputstream when finished.

    Luck,
    CJSL
    Chris S.
    Difficult? This is Mission Impossible, not Mission Difficult. Difficult should be easy.

  5. #5
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    My guess is an exception is being thrown that you are not handling. If the socket disconnects mid transfer (which can happen), or if you get a file write error (which can also happen), the code you have here will result in a partial file.

Similar Threads

  1. Fast Data Transfer 0.9.1
    By Java Tip in forum Java Software
    Replies: 0
    Last Post: 04-24-2008, 06:41 PM
  2. Fast Data Transfer 0.8.5
    By JavaBean in forum Java Software
    Replies: 0
    Last Post: 11-12-2007, 06:11 PM
  3. Fast Data Transfer 0.6.7
    By Jamie in forum Java Software
    Replies: 0
    Last Post: 06-14-2007, 02:55 PM
  4. Fast Data Transfer 0.6.4
    By levent in forum Java Software
    Replies: 0
    Last Post: 05-21-2007, 10:11 AM
  5. Fast Data Transfer 0.6.4
    By levent in forum Java Software
    Replies: 0
    Last Post: 05-20-2007, 08:55 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •