Results 1 to 10 of 10
  1. #1
    ONS
    ONS is offline Member
    Join Date
    Oct 2008
    Posts
    4
    Rep Power
    0

    Question Cleaning txt file

    I need some advise on a direction. I have a file (log file from application) that I need to "scrub". Basically I need to evaluate each line in the file against a list of strings and remove the entire line then write it to another file.

    I am not asking for anyone to write the code for me. I am only asking for a direction.

    Any thoughts or ideas?

    ONS
    Last edited by ONS; 10-08-2008 at 05:49 PM.

  2. #2
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,406
    Rep Power
    25

    Default

    To remove text from a file, the file has to be rewritten with the desired content.
    One way is to read the original file and copy to another work file what you want to save. At EOF, rename/delete the original file and rename the work file to the original name.

  3. #3
    Eranga's Avatar
    Eranga is offline Moderator
    Join Date
    Jul 2007
    Location
    Colombo, Sri Lanka
    Posts
    11,372
    Blog Entries
    1
    Rep Power
    20

    Default

    Quote Originally Posted by ONS View Post
    I am not asking for anyone to write the code for me. I am only asking for a direction.

    Any thoughts or ideas?

    ONS
    Really nice. :)

    Here is another suggestion. First read the file name and create new file with that name. Then read a line by line from the original file, and do the process on each. Finally delete the original file. To read and write you can use many ways.

  4. #4
    ONS
    ONS is offline Member
    Join Date
    Oct 2008
    Posts
    4
    Rep Power
    0

    Default

    Thank you both for the response. I know my post was vague and apologize. The direction I am looking for is more about evaluating the line of data. I have a list of 80 words or phrases that I need to remove from the file. Not only remove from the file but remove the entire line that the word or phrase resides. I would prefer not to hard code each of these as there will come a day when I need to add or delete one.

    ONS

  5. #5
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,406
    Rep Power
    25

    Default

    To see if a line contains a string, see the indexOf method.
    Put the list of words in a file and read the file.

  6. #6
    ONS
    ONS is offline Member
    Join Date
    Oct 2008
    Posts
    4
    Rep Power
    0

    Default Cleaning txt file

    Thank you!

    I have most of my code written. I am using the indexOf to find the string and I am reading the pattern matching strings into an array. I am assuming that there is no easy way to pattern match these. If I have an log file that is 40,000 lines and a pattern matching file that has 100 lines, I will end up reading up to 4,000,000 times.

    In short, I will have to read the 100 line pattern file for each of the 40,000 lines.

    Here is my unfinished code. I have code in there just to test parts.

    Again, I am not looking for someone to write the code for me just get me going in the right direction.

    Java Code:
    import java.io.*;
    
    class copyfile_1 {
        public static void main(String[] args) {
            String [] aPattern = new String[10];
            int count = 0;
    
            try {
                File inputFile 	= new File(args[0]);
                File outputFile = new File(args[1]);
                File patternFile = new File(args[2]);
    
    			BufferedReader fis = new BufferedReader(new FileReader(inputFile));
    			BufferedWriter fos = new BufferedWriter(new FileWriter(outputFile));
    			BufferedReader pis = new BufferedReader(new FileReader(patternFile));
    
                String line = null;
                String pline = null;
                String Pattern = "SELECT ";
    
    			while ((pline = pis.readLine ()) != null)	// Read in pattern matching file
    			{
    				aPattern[count] = pline;
    				System.out.println(pline);
    			}
    
                while ((line = fis.readLine()) != null)		// Read in source file
                {
    
    				if (line.indexOf(Pattern) > 0 )			// Pattern matching
    					{
    	              	fos.write(line);
    	              	fos.newLine();
    					}
    
    	            else
    	            	{
    	            	fos.write("");
    					}
                }
    
                fis.close();
                fos.close();
            } catch (FileNotFoundException e) {
                System.err.println("FileStreamsTest: " + e);
            } catch (IOException e) {
                System.err.println("FileStreamsTest: " + e);
            }
        }
    }
    ONS

  7. #7
    Supamagier is offline Senior Member
    Join Date
    Aug 2008
    Posts
    384
    Rep Power
    7

    Default

    Because I'm in a good mood:
    Java Code:
    public final boolean removeFromFile(final String file, final String toRemove, boolean start) {
            if (isinFile(file, toRemove)) {
                try {
                    File temp = new File(file+".tmp");
                    File file1 = new File(file);
                    BufferedReader reader = new BufferedReader(new FileReader(file1));
                    BufferedWriter writer = new BufferedWriter(new FileWriter(temp));
                    String line;
                    while ((line = reader.readLine()) != null && line.trim().length() > 0)
                        if ((!start && !line.equalsIgnoreCase(toRemove)) && !isinString(line,toRemove) || (start && !line.startsWith(toRemove)))
                            writer.write(line+"\n");
                        else if (!start && isinString(line,toRemove))
                            writer.write(removeFromString(line,toRemove)+"\n");
                    reader.close();
                    writer.flush();
                    writer.close();
                    if (!file1.delete()) {
                        echo("Error deleting file "+file1);
                        return false;
                    }
                    if (!temp.renameTo(file1)) {
                        echo("Error renaming file "+temp+" to file "+file1);
                        return false;
                    }
                }
                catch (Exception e) {
                    echo("Error in boolean removeFromFile(String, String, boolean): "+e);
                    e.printStackTrace();
                    return false;
                }
                return true;
            }
            return false;
        }
    I wrote a small library once to manipulate files and such. So... here is the method to remove a String from a file. You might find this useful.
    I die a little on the inside...
    Every time I get shot.

  8. #8
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,406
    Rep Power
    25

    Default

    have to read the 100 line pattern file for each of the 40,000 lines.
    Doesn't make sense.
    Why not read the 100 patterns one time and save them in a collection before trying to match them against the records in the log file.

  9. #9
    ONS
    ONS is offline Member
    Join Date
    Oct 2008
    Posts
    4
    Rep Power
    0

    Default

    I started reading about collection and it seems that it will work better than my original plan. This is the first I have worked with collection and will take me a little time to digest.

    ONS

  10. #10
    Norm's Avatar
    Norm is online now Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,406
    Rep Power
    25

    Default

    I used the term collection in a generic sense. It includes things like an array or a Vector besides the newer classes such as ArrayList.

Similar Threads

  1. Replies: 2
    Last Post: 04-07-2009, 06:25 AM
  2. Cleaning out the closet
    By Norm in forum Java Software
    Replies: 6
    Last Post: 10-31-2008, 12:18 PM
  3. code for cleaning variables?
    By Ak-Emm in forum New To Java
    Replies: 13
    Last Post: 08-22-2008, 02:40 AM
  4. To open an image file such as Jpeg file using JAva Program
    By itmani2020 in forum Advanced Java
    Replies: 10
    Last Post: 07-11-2008, 09:57 AM
  5. Replies: 7
    Last Post: 05-23-2008, 03:46 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •