Stuck with my program FTP upload, download + encryp,decrypt

[jurka]

I am really stuck with my first bigger project. So the main thing what program should do is to encrypt file-> upload. decrypt -> download. But yes FTP part works great download and upload, but when downloading from server and start decrypt i loose bytes or gets to much bytes.

Ill appreciate any help, posts, ideas how to make different. Really stuck with this messing like 1 week with this. Tried to TDD some parts but dont know how to fix.

Here encrypt - decrypt works when use in local machine

package ftp;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.spec.AlgorithmParameterSpec;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.KeySpec;

import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.CipherOutputStream;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.PBEParameterSpec;

public class Crypt {
	private final static int BUFFER_SIZE = 1024;
	private Cipher ecipher;
	private Cipher dcipher;
	private byte[] buf = new byte[BUFFER_SIZE];
	
	// 8-byte Salt
	byte[] salt = { (byte) 0xA9, (byte) 0x9B, (byte) 0xC8, (byte) 0x32,
			(byte) 0x56, (byte) 0x35, (byte) 0xE3, (byte) 0x03 };

	// Iteration count
	int iterationCount = 19;

	public Crypt(String passPhrase) throws NoSuchAlgorithmException, NoSuchPaddingException, 
		InvalidKeyException, InvalidAlgorithmParameterException, InvalidKeySpecException {
		// Create the key
		KeySpec keySpec = new PBEKeySpec(passPhrase.toCharArray(), salt,
				iterationCount);
		SecretKey key = SecretKeyFactory.getInstance("PBEWithMD5AndDES")
				.generateSecret(keySpec);
		ecipher = Cipher.getInstance(key.getAlgorithm());
		dcipher = Cipher.getInstance(key.getAlgorithm());

		// Prepare the parameter to the ciphers
		AlgorithmParameterSpec paramSpec = new PBEParameterSpec(salt,
				iterationCount);

		// Create the ciphers
		ecipher.init(Cipher.ENCRYPT_MODE, key, paramSpec);
		dcipher.init(Cipher.DECRYPT_MODE, key, paramSpec);
	}
	
	public void encrypt(InputStream in, OutputStream out) throws Exception {
		out = new CipherOutputStream(out, ecipher);
		
		int numRead = 0;
		while ((numRead = in.read(buf)) != -1) {
			out.write(buf, 0, numRead);
		}
		in.close();
		out.close();
	}

	public void decrypt(InputStream in, OutputStream out) throws Exception {
		in = new CipherInputStream(in, dcipher);
		
		int numRead = 0;
		while ((numRead = in.read(buf)) != -1) {
			out.write(buf, 0, numRead);
		}
		in.close();
		out.close();
	}

}

Example how it works

 import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.spec.InvalidKeySpecException;
 
import javax.crypto.NoSuchPaddingException;
 
import junit.framework.TestCase;
 

public class TestCryption extends TestCase {
 
	File file = new File("C:\\test.txt");
	
	public void testEncryption() {
		try {
			Crypt c = new Crypt("kass");
			c.encrypt(new FileInputStream("C:\\winhelp.exe"), new FileOutputStream("C:\\winhelp2.exe"));
			
			c.decrypt(new FileInputStream("C:\\winhelp2.exe"), new FileOutputStream("c:\\decrypted.exe"));
		} catch (InvalidKeyException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (NoSuchAlgorithmException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (NoSuchPaddingException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (InvalidAlgorithmParameterException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (InvalidKeySpecException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (Exception e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
	}
}

Here is my FTPLayer

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPConnectionClosedException;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPReply;

public class FTPLayer extends FTPClient {
	private Config c;

	
	public boolean connectAndLogin() throws IOException,
			UnknownHostException, FTPConnectionClosedException {

		String workingDir = System.getProperty("user.dir");
		String configFile = workingDir + "\\config.txt";
		boolean success = false;
		
		c = new Config(new File(configFile));
		
		connect(c.getValue("server"), Integer.parseInt(c.getValue("port")));
		int reply = getReplyCode();

		if (FTPReply.isPositiveCompletion(reply))
			success = login(c.getValue("username"), c.getValue("password"));

		if (!success) {
			disconnect();
			return false;
		}

		return success;
	}

	public void setPassiveMode(boolean setPassive) {
		if (setPassive)
			enterLocalPassiveMode();
		else
			enterLocalActiveMode();
	}

	public boolean ascii() throws IOException {
		return setFileType(FTP.ASCII_FILE_TYPE);
	}

	public boolean binary() throws IOException {
		return setFileType(FTP.BINARY_FILE_TYPE);
	}

	public boolean downloadFile(String serverFile, String localFile)
			throws IOException, FTPConnectionClosedException {

		FileOutputStream out = new FileOutputStream(localFile);
		boolean result = retrieveFile(serverFile, out);
		out.close();
		return result;
	}

	public InputStream downloadFile(String serverFile) throws IOException,
			FTPConnectionClosedException {
		return retrieveFileStream(serverFile);
	}

	public boolean uploadFile(File localfile, String serverFile)
			throws IOException, FTPConnectionClosedException {

		// OutputStream output = storeFileStream(localFile);
		// File file = new File(localFile);

		if (!localfile.exists())
			throw new FileNotFoundException();

		System.out.println("File location: " + localfile.getAbsolutePath());

		FileInputStream in = new FileInputStream(localfile.getAbsolutePath());

		boolean result = storeFile(serverFile, in);
		in.close();

		return result;
	}

	public OutputStream uploadFile(String serverFile) throws IOException,
			FTPConnectionClosedException {

		return storeFileStream(serverFile);
	}

	public List<String> listFileNames() throws IOException,
			FTPConnectionClosedException {
		FTPFile[] files = listFiles();
		List<String> v = new ArrayList<String>();

		for (int i = 0; i < files.length; i++) {
			if (!files[i].isDirectory())
				v.add((String) files[i].getName());
		}

		return v;
	}

	// public String listFileNamesString()
	// throws IOException, FTPConnectionClosedException {
	// return vectorToString(listFileNames(), "\n");
	// }

	public List<String> listSubdirNames() throws IOException,
			FTPConnectionClosedException {

		FTPFile[] files = listFiles();
		List<String> v = new ArrayList<String>();
		for (int i = 0; i < files.length; i++) {
			if (files[i].isDirectory()) {
				v.add(files[i].getName());
			}
		}
		return v;
	}

	// public String listSubdirNamesString()
	// throws IOException, FTPConnectionClosedException {
	// return vectorToString(listSubdirNames(), "\n");
	// }

//	private String vectorToString(Vector v, String delim) {
//		StringBuffer sb = new StringBuffer();
//		String s = "";
//		for (int i = 0; i < v.size(); i++) {
//			sb.append(s).append((String) v.elementAt(i));
//			s = delim;
//		}
//
//		return sb.toString();
//	}
	
	@Override
	public void disconnect() throws IOException {
		if (FTPReply.isPositiveCompletion(getReplyCode()))  {
			super.disconnect();
		}
		
	}
}

Upload with the encryption

   						FTPLayer f = new FTPLayer();
    						String filename = addCustomWordInFile(filenameText.getText(), authorText.getText());
    						
    						f.connectAndLogin();
    						System.out.println(f.getReplyString());
    						
    						f.setPassiveMode(true);
    						f.binary();
    						
    						Crypt c = new Crypt(passwordField.getText());
    						OutputStream out = f.storeFileStream(filename);
    						FileInputStream in = new FileInputStream(file);
    						
    						c.encrypt(in, out);
//    						
//							Crypt crypt = new Crypt(passwordField.getText());	
//							crypt.encrypt(in, out);
//
							ftp.disconnect();

downloading part

                                	FTPLayer f = new FTPLayer();
                                	
                        			f.connectAndLogin();
                        			f.setPassiveMode(true);
                        			f.binary();
                        		  int buffer = 0;
//                                    while ((buffer = in.read()) != -1) {
//                                        out.write(buffer);
//                                 	
                        			InputStream ins = f.retrieveFileStream(file.getName());
                        			FileOutputStream outs = new FileOutputStream(downlaodFileLocation + File.separator + file.getName());
                        			Crypt c2 = new Crypt(passwordField.getText());
                        			c2.decrypt(ins, outs);

Testing local encrypt decrypt

package test;

import static org.junit.Assert.*;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;

import org.junit.Test;

import ftp.Crypt;


public class LocalEncrypDecrypt {
	
	String[] testFilenames = { "calc", "charmap", "clipbrd", "mshearts" };
	String password = "newPass";
	
	String path = "C:\\WINDOWS\\system32\\";
	String newPath = "C:\\";
	String extension = ".exe";
	
	@Test
	public void defaultFileIsEqualWithDecrypted() throws Exception {
		Crypt c = new Crypt(password);
		
		for (String name : testFilenames) {
			c.encrypt(new FileInputStream(path + name + extension), 
					new FileOutputStream(newPath + name + "-encrypt" + extension));
			c.decrypt(new FileInputStream(newPath + name + "-encrypt" + extension), 
					new FileOutputStream(newPath + name + "-decrypt" + extension));
			
			System.out.println(name + " OK");
			
			long originalFileSize = new File(path + name + extension).length();
			long decryptedFileSize = new File(newPath + name + "-decrypt" + extension).length();
			
			assertEquals(originalFileSize, decryptedFileSize);			
		}

	}
}

[Norm]

Have you isolated the problem?
Do the encrypt/decrypt programs work when used on any file in a standalone test program?
Does the FTP program work on all types files that are not encrypted? ie images and text, binary and ASCII

If yes to both, what is different about FTPing an encrypted file?
What exactly is lost or added when doing the encrypt/decrypt.
Give an example of the data before being sent and after being received to show exactly what happens.

Are there any differences between the defaults and properties on the server vs the client?

[jurka]

I dont know how to well explain problem, but for clarification i wrote a test. And i use mostly to transfer .xsl .doc and some other doc files so i should use binary mode. Without the right file size i cant open the document.

2, 4, 5 test fail, you can use any file to test transfer

import static org.junit.Assert.assertEquals;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;

import junit.framework.Assert;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;



public class FTPencryptDecrypt {
	FTPLayer ftp = new FTPLayer();
	String filename = "C:\\winhelp.exe";
	String encryptedFilename = "C:\\winhelp-encrypted.exe";
	String encryptedFilenameLocal = "C:\\winhelp-encrypted-local.exe";
	String decrypted = "C:\\winhelp-dec.exe";
	String password = "pass";
	String ftpFilename = "winhelp-ftp.exe";
	
	@Test
	public void encryptionWithoutUploading() throws Exception {
		Crypt c = new Crypt(password);
		c.encrypt(new FileInputStream(filename), 
				new FileOutputStream(encryptedFilename));
		

	}

	@Test
	public void decryptLocal() throws Exception {
		/*
		 * This one fails
		 */
		Crypt c = new Crypt(password);
		c.decrypt(new FileInputStream(encryptedFilename),
				new FileOutputStream(decrypted));
		
		assertEquals(new File(decrypted).length(), 
				new File(filename).length());
	}
	
	@Test
	public void ftpUploadEncryption() throws Exception {
		///FTP
		ftp.connectAndLogin();
		ftp.setPassiveMode(true);
		ftp.binary();
		
		Crypt c = new Crypt(password);
		c.encrypt(new FileInputStream(encryptedFilename), 
				ftp.storeFileStream(ftpFilename));
		
//		byte[] buffer = new byte[1024];
		
//		int bytesRead = 0;
//		while ((bytesRead = in.read(buffer)) != -1) {
//			out.write(buffer);
//		}
		
//		in.close();
//		out.close();
		
		ftp.disconnect();
	}

	@Test
	public void LocalAndFTPFileSizesAreEqual() throws Exception {
		ftp.connectAndLogin();
		ftp.setPassiveMode(true);
		ftp.binary();
		
		InputStream in = ftp.retrieveFileStream(ftpFilename);
		FileOutputStream out = new FileOutputStream(encryptedFilenameLocal);
		
		byte[] buffer = new byte[1024];
		int bytesRead = 0;
		while ((bytesRead = in.read()) != -1) {
			out.write(bytesRead);
		}
		
		in.close();
		out.close();
		
		long localFileSize = new File(encryptedFilename).length();
		long encryptedFilenameLocalSize = new File(encryptedFilenameLocal).length();
		
		System.out.println("Encrypted file local: " + localFileSize);
		System.out.println("Encrypted file downlaoded from FTP: " + encryptedFilenameLocalSize);		
		assertEquals(localFileSize, encryptedFilenameLocalSize);
	}
	
	
	@Test
	public void DecryptLocalFile() throws Exception {
		Crypt c = new Crypt(password);
		c.decrypt(new FileInputStream(encryptedFilenameLocal), 
				new FileOutputStream("C:\\winhelp-decrypted.exe"));
		
//		in.close();
//		out.close();

		long decryptedLocal = new File("C:\\winhelp-decrypted.exe").length();
		long decryptedUpload = new File(filename).length();
		
		System.out.println("Decrypted Size local: " + decryptedLocal);
		System.out.println("Decrypted Size uploaded: " + decryptedUpload);
		assertEquals(decryptedLocal, decryptedUpload);
		
	}

}

[Norm]

1)Do the encrypt/decrypt programs work when used on any file in a standalone test program?
2)Does the FTP program work on all types files that are not encrypted? ie images and text, binary and ASCII

What are the answers to questions 1 & 2?

For system testing to find the problem you need a small file of test data that you can look at with a compare program or hex-editor that will compare the files byte by byte.

Without the right file size i cant open the document.

What does this mean? What program are you using to open the file?

[jurka]

Wrote new test case for standalone programs encrypt/decrypt testing. Also you can run it to when got xp. Passed this test, but needed to modify the Crypt class. Now the byte difference is smaller.

2) yes it works.

package test;

import static org.junit.Assert.*;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;

import org.junit.Test;

import ftp.Crypt;


public class LocalEncrypDecrypt {
	
	String[] testFilenames = { "calc", "charmap", "clipbrd", "mshearts" };
	String password = "newPass";
	
	String path = "C:\\WINDOWS\\system32\\";
	String newPath = "C:\\";
	String extension = ".exe";
	
	@Test
	public void defaultFileIsEqualWithDecrypted() throws Exception {
		Crypt c = new Crypt(password);
		
		for (String name : testFilenames) {
			c.encrypt(new FileInputStream(path + name + extension), 
					new FileOutputStream(newPath + name + "-encrypt" + extension));
			c.decrypt(new FileInputStream(newPath + name + "-encrypt" + extension), 
					new FileOutputStream(newPath + name + "-decrypt" + extension));
			
			System.out.println(name + " OK");
			
			long originalFileSize = new File(path + name + extension).length();
			long decryptedFileSize = new File(newPath + name + "-decrypt" + extension).length();
			
			assertEquals(originalFileSize, decryptedFileSize);			
		}

	}
}

[Norm]

Still trying to isolate the problem. You say that:
1) The encrypt/decrypt program has been tested and works with all kinds of input (binary and text). The output matches the input exactly!!
2)The FTP program works will all kinds of files. Again the output matches the input exactly.

If the FTP program works with all kinds of files, what fails when the input to the FTP program is an encrypted file?
Can you create an encrypted file and send it with the FTP program and see if the file is transmitted correctly? Does what the server receive match exactly the file that was sent?
If there is a problem then #2 above is wrong because you've said that the FTP program works for all files.

The test program you show above is too simple. It only checks to see if the lengths are the same, not that ALL of the contents are the same. Read and compare all the bytes if the lengths are the same.

[jurka]

ill try to do some problem isolating. But atm when i used plain file to download and upload it worked.

[Norm]

So you're not sure that the programs work with binary data. Which part encrypt/decrypt or FTP or both?

For testing:
Write a one page program to create a test file with 256 bytes with values from 0 to 255. Use FileOutputStream and write(). No GUI needed just a loop to write 256 bytes.
Then another program to read two files using FileInputStreams and have a loop to read the bytes and compare them one by one. Again no GUI needed, just hardcode the file names in the program. Print out the mismatches.
For the above test file the value of the input file will be the index into the file. ie byte[0] = 0, etc
For other files, output the index as well as the values in both files that were different.

[fishtoprecords]

You really need to separate your problems. Crypto can be very hard to debug.

Change your enciper/decipher function to a no-op.
Verify that the files are transfered exactly. Use
shasum or md5sum to prove that they are byte identical.

Remember, when crypto works right, it takes good stuff and turns it into random looking garbage. When it works wrong, it takes good stuff and turns it into random looking garbage. You can't tell if its correct by looking at it.

You must be able to test the crypto code separately from all else.