Results 1 to 8 of 8
Like Tree1Likes
  • 1 Post By Junky

Thread: Tokenizer

  1. #1
    skaterboy987 is offline Member
    Join Date
    Sep 2011
    Posts
    56
    Rep Power
    0

    Default Tokenizer

    I'm writing a program to reverse whatever words you enter... Here is the result I would like to have

    My input: input1 input2 input3
    Wanted Output:
    input1
    input2
    input3


    Yes I have looked at the person's topic who was attempting the same thing, but it didn't really help for my problem
    Here's my result with what I have:

    My input:input1 input2 input3
    output: null
    output: input3
    output: input2
    output: Exception in thread "main" java.lang.NullPointerException
    at Token.main(Token.java:23)


    Java Code:
    import java.util.Scanner;
    import java.util.StringTokenizer;
    
    
    public class Token {
    
    	 public static void main(String args[]) {
    		 int i = 0;
    		 String[] token;              // declares an array of integers
             token = new String[5];      // allocates memory for 10 integers
             Scanner scanner = new Scanner(System.in);
            String text = scanner.nextLine();
        StringTokenizer st = new StringTokenizer(text);
     
        while (st.hasMoreTokens()) 
        {
            i++;
    		token[i] = st.nextToken();
    		
     }
    	for(int x=1; x<=(token.length- 1); x++)
    	{
    		if (token[x].equalsIgnoreCase("null") ) {
    		}
    		else
    		{
    		System.out.println(token[token.length - x]);
    		}
    	}
    }
    }

  2. #2
    Fubarable's Avatar
    Fubarable is offline Moderator
    Join Date
    Jun 2008
    Posts
    19,316
    Blog Entries
    1
    Rep Power
    26

    Default Re: Tokenizer

    Use println's to tell you what's going on:

    Java Code:
        while (st.hasMoreTokens()) 
        {
            i++;
            token[i] = st.nextToken();
            System.out.println("i is: " + i + "; and token[i] is: " + token[i]);
         }
    This may give you a hint as to what your problem is (hint look carefully at i's value).

  3. #3
    Fubarable's Avatar
    Fubarable is offline Moderator
    Join Date
    Jun 2008
    Posts
    19,316
    Blog Entries
    1
    Rep Power
    26

    Default Re: Tokenizer

    Also:

    Java Code:
    public class Token {
    
    	 public static void main(String args[]) {
    		 int i = 0;
    		 String[] token;              // declares an array of integers  *** WTF? ***
             token = new String[5];      // allocates memory for 10 integers  *** WTF? ***
             Scanner scanner = new Scanner(System.in);
            String text = scanner.nextLine();
        StringTokenizer st = new StringTokenizer(text);
     
        while (st.hasMoreTokens()) 
        {
            i++;
    		token[i] = st.nextToken();
    		
     }
    	for(int x=1; x<=(token.length- 1); x++)
    	{
    		if (token[x].equalsIgnoreCase("null") ) {   // *** WTF? ***
    		}
    		else
    		{
    		System.out.println(token[token.length - x]);
    		}
    	}
    }
    }
    • Use String#split(...) if you can instead of a StringTokenizer. That way you'll be able to create an array of the right size no matter what text is entered
    • Be sure your comments agree with your code!
    • Never check for null that way lest you want to have lots of NPE's thrown. You would do it as if (token[i] == null) {
    • Please be compulsive about using proper code indentation and consistent formatting, including curly brace formatting. Coding is an exercise in precision and there's no place in it for sloppiness. This will also cut down on the number of code errors you produce.

  4. #4
    kennyman94 is offline Member
    Join Date
    Feb 2011
    Posts
    24
    Rep Power
    0

    Default Re: Tokenizer

    why not just read the entire text file into one string, call its split(String regex) method, and then iterate over the returned array in a reversed for-loop (start at a number then decrement to 0)? if you want to seperate each token by whitespace, then just call:
    String[] tokens = text.split(" ");
    Last edited by kennyman94; 10-21-2011 at 04:16 AM. Reason: forgot to mention something

  5. #5
    Junky's Avatar
    Junky is offline Grand Poobah
    Join Date
    Jan 2011
    Location
    Dystopia
    Posts
    3,798
    Rep Power
    7

    Default Re: Tokenizer

    Quote Originally Posted by kennyman94 View Post
    why not just read the entire text file into one string
    Not a good idea when dealing with large files. You will run out of memory.
    Fubarable likes this.

  6. #6
    kennyman94 is offline Member
    Join Date
    Feb 2011
    Posts
    24
    Rep Power
    0

    Default Re: Tokenizer

    I would think it would be pretty hard to run out of memory with a String. the JFugue api uses Strings to represent notes, chords, durations, etc. for midi and it can use Miles of characters in a single String (from what i hear anyways)

  7. #7
    Junky's Avatar
    Junky is offline Grand Poobah
    Join Date
    Jan 2011
    Location
    Dystopia
    Posts
    3,798
    Rep Power
    7

    Default Re: Tokenizer

    I have a crappy application at work that reads files and is constantly running out of memory when it tries to read large files.

  8. #8
    kennyman94 is offline Member
    Join Date
    Feb 2011
    Posts
    24
    Rep Power
    0

Similar Threads

  1. String Tokenizer help
    By GreenTea in forum New To Java
    Replies: 4
    Last Post: 10-30-2010, 02:44 AM
  2. String Tokenizer
    By sumaih in forum Java Gaming
    Replies: 2
    Last Post: 08-21-2010, 03:23 PM
  3. String Tokenizer
    By hussainian in forum Advanced Java
    Replies: 1
    Last Post: 03-16-2010, 08:58 AM
  4. String Tokenizer
    By viperlasson in forum New To Java
    Replies: 1
    Last Post: 03-09-2010, 01:14 PM
  5. String Tokenizer
    By redasu in forum Advanced Java
    Replies: 4
    Last Post: 02-19-2010, 03:30 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •