Results 1 to 2 of 2
  1. #1
    Join Date
    Mar 2012
    Rep Power

    Default Can StreamTokenizer find numbers followed by letters

    I have an assignment where I have to write 2 lexers for a grammar given to me by my teacher.

    First we need to use Streamtokenizer, Stringtokenizer or Scanner. The second uses regular expressions. The second I have working fine. The first doesn't work properly.

    Basically I need to find all tokens that start with letters and then are followed by any number of letters and numbers.

    I have a test file:

    INT f2(INT x, INT y )
    z := x*x - y*y;
    RETURN z;
    INT MAIN f1()

    INT x;
    READ(x, "A41.input");
    INT y;
    READ(y, "A42.input");
    INT z; // sdfsdf';lsdf
    z := f2(x,y) + f2(y,x);
    WRITE (z, "A4.output");


    So it is supposed to find z,x,y,f1,f2... but not 1234sdfas. And streamtokenizer finds sdfas. Is there anyway to prevent it from doing that. maybe checking to see if the number is followed by a space? Keep in mind i'm not allowed to use any regular expressions in this example.

    I'm not sure how to create a lookahead token because it seems to eat another token and I don't want that. Unless there is a way to use the next token and the pushback() method. Any help would be appreciated.

  2. #2
    quad64bit's Avatar
    quad64bit is offline Moderator
    Join Date
    Jul 2009
    Rep Power

    Default Re: Can StreamTokenizer find numbers followed by letters

    Someone is taking compiler!

    If I understand this example correctly, shouldn't 1234sdfas be a syntax error? If so, you would want your scanner to faithfully read this, and your parser should reject it once it attempts to reduce it from the stack as a syntax error.

    When I wrote my compiler, the only thing stripped out at the scanner level was whitespace and comments (line and block).

    A lookahead simply means you always read one char ahead, meaning you need to always store the current token and the next one, but only look at the next one when you need to do a look-ahead, or if you are finished with the current token, and you shift the look ahead into the current position, and read a new lookahead from the file.

Similar Threads

  1. Replies: 2
    Last Post: 12-09-2011, 02:57 AM
  2. Replies: 2
    Last Post: 11-30-2011, 04:41 PM
  3. Stop user entering letters in GUI, numbers only
    By africanhacker in forum New To Java
    Replies: 3
    Last Post: 03-11-2011, 01:25 PM
  4. converting letters to numbers
    By fenhoff in forum New To Java
    Replies: 2
    Last Post: 01-24-2010, 04:45 AM
  5. validating a string for numbers and letters?
    By lockmac in forum New To Java
    Replies: 1
    Last Post: 08-09-2007, 09:17 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts