Cleaning txt file

[ONS]

I need some advise on a direction. I have a file (log file from application) that I need to "scrub". Basically I need to evaluate each line in the file against a list of strings and remove the entire line then write it to another file.

I am not asking for anyone to write the code for me. I am only asking for a direction.

Any thoughts or ideas?

ONS

[Norm]

To remove text from a file, the file has to be rewritten with the desired content.
One way is to read the original file and copy to another work file what you want to save. At EOF, rename/delete the original file and rename the work file to the original name.

[Eranga]

I am not asking for anyone to write the code for me. I am only asking for a direction.

Any thoughts or ideas?

ONS

Really nice. :)

Here is another suggestion. First read the file name and create new file with that name. Then read a line by line from the original file, and do the process on each. Finally delete the original file. To read and write you can use many ways.

[ONS]

Thank you both for the response. I know my post was vague and apologize. The direction I am looking for is more about evaluating the line of data. I have a list of 80 words or phrases that I need to remove from the file. Not only remove from the file but remove the entire line that the word or phrase resides. I would prefer not to hard code each of these as there will come a day when I need to add or delete one.

ONS

[Norm]

To see if a line contains a string, see the indexOf method.
Put the list of words in a file and read the file.

[ONS]

Thank you!

I have most of my code written. I am using the indexOf to find the string and I am reading the pattern matching strings into an array. I am assuming that there is no easy way to pattern match these. If I have an log file that is 40,000 lines and a pattern matching file that has 100 lines, I will end up reading up to 4,000,000 times.

In short, I will have to read the 100 line pattern file for each of the 40,000 lines.

Here is my unfinished code. I have code in there just to test parts.

Again, I am not looking for someone to write the code for me just get me going in the right direction.

import java.io.*;

class copyfile_1 {
    public static void main(String[] args) {
        String [] aPattern = new String[10];
        int count = 0;

        try {
            File inputFile 	= new File(args[0]);
            File outputFile = new File(args[1]);
            File patternFile = new File(args[2]);

			BufferedReader fis = new BufferedReader(new FileReader(inputFile));
			BufferedWriter fos = new BufferedWriter(new FileWriter(outputFile));
			BufferedReader pis = new BufferedReader(new FileReader(patternFile));

            String line = null;
            String pline = null;
            String Pattern = "SELECT ";

			while ((pline = pis.readLine ()) != null)	// Read in pattern matching file
			{
				aPattern[count] = pline;
				System.out.println(pline);
			}

            while ((line = fis.readLine()) != null)		// Read in source file
            {

				if (line.indexOf(Pattern) > 0 )			// Pattern matching
					{
	              	fos.write(line);
	              	fos.newLine();
					}

	            else
	            	{
	            	fos.write("");
					}
            }

            fis.close();
            fos.close();
        } catch (FileNotFoundException e) {
            System.err.println("FileStreamsTest: " + e);
        } catch (IOException e) {
            System.err.println("FileStreamsTest: " + e);
        }
    }
}

ONS

[Supamagier]

Because I'm in a good mood:

public final boolean removeFromFile(final String file, final String toRemove, boolean start) {
        if (isinFile(file, toRemove)) {
            try {
                File temp = new File(file+".tmp");
                File file1 = new File(file);
                BufferedReader reader = new BufferedReader(new FileReader(file1));
                BufferedWriter writer = new BufferedWriter(new FileWriter(temp));
                String line;
                while ((line = reader.readLine()) != null && line.trim().length() > 0)
                    if ((!start && !line.equalsIgnoreCase(toRemove)) && !isinString(line,toRemove) || (start && !line.startsWith(toRemove)))
                        writer.write(line+"\n");
                    else if (!start && isinString(line,toRemove))
                        writer.write(removeFromString(line,toRemove)+"\n");
                reader.close();
                writer.flush();
                writer.close();
                if (!file1.delete()) {
                    echo("Error deleting file "+file1);
                    return false;
                }
                if (!temp.renameTo(file1)) {
                    echo("Error renaming file "+temp+" to file "+file1);
                    return false;
                }
            }
            catch (Exception e) {
                echo("Error in boolean removeFromFile(String, String, boolean): "+e);
                e.printStackTrace();
                return false;
            }
            return true;
        }
        return false;
    }

I wrote a small library once to manipulate files and such. So... here is the method to remove a String from a file. You might find this useful.

[Norm]

have to read the 100 line pattern file for each of the 40,000 lines.

Doesn't make sense.
Why not read the 100 patterns one time and save them in a collection before trying to match them against the records in the log file.

[ONS]

I started reading about collection and it seems that it will work better than my original plan. This is the first I have worked with collection and will take me a little time to digest.

ONS

[Norm]

I used the term collection in a generic sense. It includes things like an array or a Vector besides the newer classes such as ArrayList.