Tokenizing with Scanner

[sireesha]

Hi all,
Please check this code..

Code:

import java.util.Scanner;
public class Scanner_Tokenizing
{
public static void main(String[] args)
{
 
    System.out.println("Enter some string:");
    Scanner s=new Scanner(System.in);
    String st;
    int count=0;
    boolean b=false;
    
   while(b=s.hasNext())
    {
        st=s.next();
        count++;
        System.out.println(st);
    }    
   System.out.println("Total number of tokens : "+count);
}        
}

While running this code..it just prints all the tokens and continuously running..
It never came to the output statement after while loop..
What's wrong with my code ?
Please tell me.
Thanq very much for reading this.

[tim]

Hello sireesha

This was very challenging for me (no IDE used ), but I have a solution!

Code:

import java.util.Scanner;
import java.io.*;
public class Scanner_Tokenizing
{
	public static void main(String[] args)	{
		try{
			System.out.println("Enter some string:");
			DataInputStream input = new DataInputStream(System.in);
			String text = input.readLine();
			Scanner scanner = new Scanner(text);
			int count = 0;
			while (scanner.hasNext() == true){
				String st = scanner.next();
				count++;
				System.out.println(st);
			}    
			System.out.println("Total number of tokens : " + count);
		} catch (Exception e){
			System.out.println(e.getMessage());
		}
	}
}

The fragments in blue are the important ones. Remember that the System.in input stream can be infinitively long. So, the scanner does not know that a new line symbol means stop. That's why I read the input stream with a DataInputStream object before hand, to obtain a finite String object to work with. Also remember that when you compare stuff, you must use "=="

I hope this helps!

[hardwired]

Code:

import java.util.*;
import java.util.regex.Pattern;

public class Test {
    public static void main(String[] args) {
        System.out.println("Enter some string:");
        Scanner s=new Scanner(System.in);
        int count=0;

        while(s.hasNext()) {
            String st = s.nextLine();
            if(st.equalsIgnoreCase("x"))
                break;
            String[] tokens = getTokens(st);
            System.out.printf("tokens = %s%n", Arrays.toString(tokens));
            count++;
        }    
        System.out.println("Total number of tokens : "+count);
        s.close();
    }

    private static String[] getTokens(String str) {
        System.out.println("str = " + str);
        Scanner scanner = new Scanner(str);
        //System.out.println("delimiter = " + scanner.delimiter());
        String[] s = new String[str.length()];
        // Capture the next word character, space or puntuation mark.
        String regex = "(?=[\\w\\s\\p{Punct}])";
        Pattern pattern = Pattern.compile(regex);
        scanner.useDelimiter(pattern);
        //System.out.println("delimiter = " + scanner.delimiter());
        int count = 0;
        while(scanner.hasNext("[\\w\\s\\p{Punct}]")) {
            s[count++] = scanner.next("[\\w\\s\\p{Punct}]");
        }
        scanner.close();
        return s;
    }
}

[tim]

Thanks hardwired. I did not think of using regular expressions.