tokens

[Gilgamesh]

Ive used StringTokenizer to take the words from a text. I used the delimeters ",", "." etc.

questions

1) I tried to define a final string DELIMETERS="!@#" (etc) but when i type

StringTokenizer (line, DELIMITERS);

though it recognizes the delimeters, it creates one token made by every line of the text document (without including the delimeters).
if you cant figure out the problem, can you please tell me if there is a different way to set the delimeters? except from this:

StringTokenizer (line, ".", ",", "?");

?

2)some texts, at the end of the line use a hiven to continue the word to the next line. what can I 'unify' these two tokens that consist one word?

[hardwired]

a different way to set the delimeters
Try including the "space" delimiter.
some texts, at the end of the line use a hiven to continue the word to the next line. what can I 'unify' these two tokens that consist one word
Do you mean to remove the hyphen and concatenate the two words together to become a single token?

import java.util.StringTokenizer;

public class TokenDelims {
    public static void main(String[] args) {
        String s = "This is#a special test-string for testing " +
                   "deliminators in a StringTokenizer";
        String delims = " #-";
        StringTokenizer st = new StringTokenizer(s, delims);
        while(st.hasMoreTokens())
            System.out.println(st.nextToken());
    }
}

[Gilgamesh]

remove the hyphen and concatenate the two words together to become a single token i mean the use of hyphens to show that a word has been broken in order to fit onto a line.

But i am thinking now that the hyphens are also used to join words together to make a compound e.g. 'left-handed'.

thats sounds difficult.. if I use the String.split () (instead of the StringTokenizer) things gonna be easier?

so how can i make this code? 'if there is a hyphen check if the syllables that the hyphen is between them (even if there is a change of line) exist as a compound word in the (arraylist/vector) dictionary and if they do not then eliminate the hyphen and unite the syllables into one word.

pain in the neck lol

ooh and there is no ignoreCase at the StringTokenizer . :-|

[hardwired]

You can save the token in a string and lowerCase it

String token = st.nextToken();
token = token.toLowerCase();

import java.util.StringTokenizer;

public class TokenDelims {
    public static void main(String[] args) {
        String s = "This is a test-string for testing delimi-\n" +
                   "nators in a StringTokenizer";
        String delims = " ";
        StringTokenizer st = new StringTokenizer(s, delims);
        while(st.hasMoreTokens()) {
            String token = st.nextToken();
            int dash = token.indexOf("-");
            if(dash != -1) {
                int newLine = token.indexOf("\n");
                if(newLine != -1) {           // hyphen
                    int length = token.length();
                    token = token.substring(0, dash) +
                            token.substring(newLine+1, length);
                }
            }
            System.out.println(token);
        }
    }
}