Tokenizer related testing
I am tokenizing java files. I have separated the java codes into keywords and identifiers(variables). Now i have to test whether the tokens generated are correct or not. It seems to be working for few lines of codes when i inspect them manually, but i am having problem when the file is very large. I cannot manually inspect large files and check every token. We could say that if the system works for smaller files then it should work for larger ones as well, but how do i know that for sure? Are there any ways to test large files?
I thank you for your effort.