Page 1 of 2 12 LastLast
Results 1 to 20 of 31
Like Tree1Likes

Thread: Scanner does not read to the end of a very long TXT file

  1. #1
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Scanner does not read to the end of a very long TXT file

    Java Code:
    import java.io.File;
    import java.io.FileNotFoundException;
    import java.util.Scanner;
    
    public class PrintALongFile{
    	
    	public static void main(String[] args) throws FileNotFoundException{
    
    		File inFile = new File("C:\\VeryLongFile.txt");
    		Scanner input = new Scanner(inFile);
    		while (input.hasNextLine()){
    			String line = input.nextLine();
    			System.out.println(line);
    		}
    		input.close();
    		
    	}
    	
    }
    When I try to read from a file that's 4,075,904 lines long and 41,646KB in size, it wont go past line 1,266,471. The above code is simplified from my actual program, which actually prints out the results to another file. However, the console(Eclipse IDE) and the output file, both show that it's stopping at the same line.

    How can I read the to the very end of my file?
    Last edited by ImTroyMiller; 07-04-2014 at 11:33 PM. Reason: 1,266,471 is the actual line it stops on

  2. #2
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    4,033
    Rep Power
    6

    Default Re: Scanner does not read to the end of a very long TXT file

    Without knowing for certain, it could be console limitation in Eclipse. Did you simply try to read all the lines and count them without printing them out?

    Regards,
    Jim
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  3. #3
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,902
    Rep Power
    25

    Default Re: Scanner does not read to the end of a very long TXT file

    it's stopping at the same line.
    What does "It's stopping" mean?
    An exception?
    hasNextLine() returns false?
    the code hangs in a method?
    or what?
    If you don't understand my response, don't ignore it, ask a question.

  4. #4
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Re: Scanner does not read to the end of a very long TXT file

    Quote Originally Posted by jim829 View Post
    Without knowing for certain, it could be console limitation in Eclipse. Did you simply try to read all the lines and count them without printing them out?

    Regards,
    Jim
    It counted 1,266,471 lines. Nothing was printed to the console except the final result and nothing was output to a file.

    Quote Originally Posted by Norm View Post
    What does "It's stopping" mean?
    An exception?
    hasNextLine() returns false?
    the code hangs in a method?
    or what?
    The out print to the console and the out put to the other file(that it's being copied to), stop before the end of the file. The program ends without any errors.

  5. #5
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,902
    Rep Power
    25

    Default Re: Scanner does not read to the end of a very long TXT file

    program ends
    Why?
    Does hasNextLine() return false?

    Another debug idea: Sum the lengths of the lines that are read and print out the value after the loop exits.
    If you don't understand my response, don't ignore it, ask a question.

  6. #6
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    4,033
    Rep Power
    6

    Default Re: Scanner does not read to the end of a very long TXT file

    I am not certain what is going on. Was the output file the same size as the original? What you may also try is to use one of the IO classes (like BufferedReader) to read the file up to the number of characters you read and then print out some lines starting there to see what is going on. Could it be that the line terminator in the file is inconsistent with the OS?

    Regards,
    Jim
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  7. #7
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Re: Scanner does not read to the end of a very long TXT file

    Quote Originally Posted by Norm View Post
    Why?
    Does hasNextLine() return false?
    It must, because the while loop ends and, right now, it prints out the sum of the lengths of the lines.

    Java Code:
    int tot = 0;
    while (input.hasNextLine(){
        String line = input.nextLine();
        tot = tot + line.length();
    }
    System.out.println(tot);
    It makes it to the System.out.println and the programs ends.

    Quote Originally Posted by Norm View Post
    Another debug idea: Sum the lengths of the lines that are read and print out the value after the loop exits.
    10714657

    Quote Originally Posted by jim829 View Post
    Could it be that the line terminator in the file is inconsistent with the OS?
    I'm not sure.

    One thing I do know though, it that inFile.length() will give me the correct length of the file.

  8. #8
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,902
    Rep Power
    25

    Default Re: Scanner does not read to the end of a very long TXT file

    10714657 / 1266471 = 8 bytes per line. Does that sound right? That wouldn't account for the line end character(s)
    If you don't understand my response, don't ignore it, ask a question.

  9. #9
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Re: Scanner does not read to the end of a very long TXT file

    Quote Originally Posted by Norm View Post
    10714657 / 1266471 = 8 bytes per line. Does that sound right? That wouldn't account for the line end character(s)
    Well, every 5th line is blank. But I still don't understand what you're referring to? Are you saying that there's a certain limit of the amount of total bytes?

  10. #10
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    Eastern Florida
    Posts
    17,902
    Rep Power
    25

    Default Re: Scanner does not read to the end of a very long TXT file

    I have no idea what the problem is. My suggestions are to get some ideas about what is happening.
    If you don't understand my response, don't ignore it, ask a question.

  11. #11
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    4,033
    Rep Power
    6

    Default Re: Scanner does not read to the end of a very long TXT file

    Did you try what I suggested in post #6 (using BufferedReader)?

    Regards,
    Jim
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  12. #12
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Re: Scanner does not read to the end of a very long TXT file

    Quote Originally Posted by jim829 View Post
    Did you try what I suggested in post #6 (using BufferedReader)?

    Regards,
    Jim
    Yes, and now it works. Thank you so much.

    I'm assuming there's a limit of some kind that Scanner has. It'd be nice to know for sure, but searching I can't find anything.

  13. #13
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    4,033
    Rep Power
    6

    Default Re: Scanner does not read to the end of a very long TXT file

    Well, I wasn't necessarily advocating using the BufferedReader in place of the Scanner. I was thinking
    it might be useful to help diagnose the problem.

    Edit: Although I figured there was another cause I did this test anyway. I created a 50Mbyte file of lines
    from between 70 and 80 chars long, terminated with a '\n' (on Win 7). Scanner had no problem reading
    the entire file. I believe there is something in your data that is causing the problem or you may be
    misinterpreting the results. It would be a worthwhile exercise for you to try to figure out what the
    problem is rather than rely on another method that appears to work.

    Regards,
    Jim
    Last edited by jim829; 07-05-2014 at 06:38 AM.
    The JavaTM Tutorials | SSCCE | Java Naming Conventions
    Poor planning on your part does not constitute an emergency on my part

  14. #14
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Re: Scanner does not read to the end of a very long TXT file

    Thank you.

    I will take a take a look into it again. I wouldn't doubt if it's some simple little logic error I've made.

  15. #15
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,224
    Rep Power
    20

    Default Re: Scanner does not read to the end of a very long TXT file

    Java Code:
    import java.io.File;
    import java.io.FileNotFoundException;
    import java.util.Scanner;
    
    public class PrintALongFile{
    	
    	public static void main(String[] args) throws FileNotFoundException{
    
    		File inFile = new File("C:\\VeryLongFile.txt");
    		Scanner input = new Scanner(inFile);
    		while (input.hasNextLine()){
    			String line = input.nextLine();
    			System.out.println(line);
    		}
    		input.close();
    		
    	}
    	
    }
    Quote Originally Posted by ImTroyMiller View Post
    The out print to the console and the out put to the other file(that it's being copied to), stop before the end of the file. The program ends without any errors.
    The above code only shows you outputting to the console.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  16. #16
    JosAH's Avatar
    JosAH is offline Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,783
    Blog Entries
    7
    Rep Power
    21

    Default Re: Scanner does not read to the end of a very long TXT file

    Are there any 'funny' characters in that file? i.e. (x < 0x20 && x != 0x09 && x != 0x0a && x != 0x0d) || x > 0x7f;

    kind regards,

    Jos
    gimbal2 likes this.
    cenosillicaphobia: the fear for an empty beer glass

  17. #17
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Re: Scanner does not read to the end of a very long TXT file

    Quote Originally Posted by Tolls View Post
    Java Code:
    import java.io.File;
    import java.io.FileNotFoundException;
    import java.util.Scanner;
    
    public class PrintALongFile{
    	
    	public static void main(String[] args) throws FileNotFoundException{
    
    		File inFile = new File("C:\\VeryLongFile.txt");
    		Scanner input = new Scanner(inFile);
    		while (input.hasNextLine()){
    			String line = input.nextLine();
    			System.out.println(line);
    		}
    		input.close();
    		
    	}
    	
    }


    The above code only shows you outputting to the console.
    "The above code is simplified from my actual program, which actually prints out the results to another file."

    Quote Originally Posted by JosAH View Post
    Are there any 'funny' characters in that file? i.e. (x < 0x20 && x != 0x09 && x != 0x0a && x != 0x0d) || x > 0x7f;

    kind regards,

    Jos
    Not with this program, but I'm having that problem with another program, however, that program actually reads to the end of the file.

    What is it that causes this? The encoding of the file?

  18. #18
    JosAH's Avatar
    JosAH is offline Moderator
    Join Date
    Sep 2008
    Location
    Voorschoten, the Netherlands
    Posts
    13,783
    Blog Entries
    7
    Rep Power
    21

    Default Re: Scanner does not read to the end of a very long TXT file

    Quote Originally Posted by ImTroyMiller View Post
    Not with this program, but I'm having that problem with another program, however, that program actually reads to the end of the file.

    What is it that causes this? The encoding of the file?
    Could be, that's why I was asking; a small program quickly tells you the answer ...

    kind regards,

    Jos
    cenosillicaphobia: the fear for an empty beer glass

  19. #19
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    12,224
    Rep Power
    20

    Default Re: Scanner does not read to the end of a very long TXT file

    Quote Originally Posted by ImTroyMiller View Post
    "The above code is simplified from my actual program, which actually prints out the results to another file."
    Oops.
    Missed that bit!
    :)

    Does your actual code do anything with the lines read in other than print them back out?

    Oh, and from the docs:
    "
    If an invocation of the underlying readable's {@link java.lang.Readable#read} method throws an {@link java.io.IOException} then the scanner assumes that the end of the input has been reached. The most recent <tt>IOException</tt> thrown by the underlying readable can be retrieved via the {@link #ioException} method.
    "
    After the loop see if you can grab the ioException that caused it to exit and print the stack trace.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  20. #20
    ImTroyMiller is offline Member
    Join Date
    Jan 2014
    Posts
    20
    Rep Power
    0

    Default Re: Scanner does not read to the end of a very long TXT file

    I got it.

    Apparently there was a strange character in the file that was just down a little ways from where it was stopping. After deleting that, it made it further but stopped again. So I searched further down, and about 20 lines down, there was another strange character. After deleting that one, the program read to the end of the file.

    Any idea on why these two character stumped the JVM?

Page 1 of 2 12 LastLast

Similar Threads

  1. Replies: 1
    Last Post: 09-20-2012, 10:54 AM
  2. Read text file with scanner class.
    By frente158 in forum New To Java
    Replies: 3
    Last Post: 03-11-2012, 02:58 PM
  3. Replies: 2
    Last Post: 01-10-2012, 02:10 PM
  4. Using Read/Write From File Using Scanner
    By javaisntcoffee123 in forum New To Java
    Replies: 4
    Last Post: 04-15-2010, 04:35 AM
  5. Replies: 0
    Last Post: 04-17-2008, 08:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •