Tokenizer with data validation for missing text

[dug]

I'm working on a project where we have to import a file to an array. The file has country, city, region, region number, and population.

I got the constructor and tokenizer to work but it also needs to do exception handling. For example, if population was blank it wouldn't try to create an object with that line.

My question is can I even use tokenizer in this case or will I need to rewrite it to use substring? I didn't think about it when I started but if the tokenizer looks for blank space then it will be trying to parse the word after the blank space (it has crashed every time I tried to input a file missing something).

[pbrockway2]

It's sort of hard to tokenise something that isn't there!

I think I would check each token expecting all 5 to be there. Two different things could go wrong: what I find doesn't match what I expect (numbers where I expect alphabetic characters or vice versa) or not enough tokens. That second case effectively finds the blanks. (There is a third possibility: too many tokens. Maybe that should be flagged as an error.)

------------------------

If the data uses some sort of fixed width format then missing fields are detectable (and you know which field is missing). In that case I probably would use substring(), trim() what it returns and check for empty or bad strings.

[dug]

Yes, it does use a fix width. I started writing it again using substring figuring tokenizer wouldn't work with a blank space. I was just hoping to avoid doing it if possible.

Now that you mention it, I didn't think of testing for blank tokens. I got the validation to work checking for strings and numbers in the wrong field, but I'll see if I can add something to check for empty tokens too.

[dug]

Got it working! I added a catch to handle a NoSuchElementException and it kept processing the rest of the file.

Thanks for the help.

[pbrockway2]

I'll see if I can add something to check for empty tokens too.

Maybe I wasn't clear - I don't think you'll find blank tokens because the tokeniser may well skip them.

What you might find is that the tokeniser only reports four tokens. In that case you know there was a blank because there should have been five tokens.

-------------------------

But, again, fixed width says "substring" to me.

[Edit] ... slow post ;(

Glad you've got it working. Catching a NoSuchElementException is one way of seeing if the data doesn't have enough tokens.