Can StreamTokenizer find numbers followed by letters
I have an assignment where I have to write 2 lexers for a grammar given to me by my teacher.
First we need to use Streamtokenizer, Stringtokenizer or Scanner. The second uses regular expressions. The second I have working fine. The first doesn't work properly.
Basically I need to find all tokens that start with letters and then are followed by any number of letters and numbers.
I have a test file:
INT f2(INT x, INT y )
z := x*x - y*y;
INT MAIN f1()
INT z; // sdfsdf';lsdf
z := f2(x,y) + f2(y,x);
WRITE (z, "A4.output");
So it is supposed to find z,x,y,f1,f2... but not 1234sdfas. And streamtokenizer finds sdfas. Is there anyway to prevent it from doing that. maybe checking to see if the number is followed by a space? Keep in mind i'm not allowed to use any regular expressions in this example.
I'm not sure how to create a lookahead token because it seems to eat another token and I don't want that. Unless there is a way to use the next token and the pushback() method. Any help would be appreciated.
Re: Can StreamTokenizer find numbers followed by letters
Someone is taking compiler!
If I understand this example correctly, shouldn't 1234sdfas be a syntax error? If so, you would want your scanner to faithfully read this, and your parser should reject it once it attempts to reduce it from the stack as a syntax error.
When I wrote my compiler, the only thing stripped out at the scanner level was whitespace and comments (line and block).
A lookahead simply means you always read one char ahead, meaning you need to always store the current token and the next one, but only look at the next one when you need to do a look-ahead, or if you are finished with the current token, and you shift the look ahead into the current position, and read a new lookahead from the file.