Results 1 to 9 of 9
  1. #1
    hedonist is offline Member
    Join Date
    Jun 2009
    Posts
    35
    Rep Power
    0

    Default Tokenizing VB code from java

    Hi,
    I am trying to tokenize Visual Basic (VB) code from java using StreamTokenizer. I am having problem tokenizing the codes because they do not have any delimiters such as ';'. What i did was, i tried checking for space as delimiter to separate different lines of codes. Using StringTokenizer is not an option.

    I was expecting the following code would separate tokens based on space (which has ascii value of 32 in integer). But it doesn't work.

    if(StreamTokenizer.ttype==32)
    //work with separated codes.

    Thank you beforehand

  2. #2
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,317
    Rep Power
    25

    Default

    StreamTokenizer.ttype==32)
    More understandable would be: StreamTokenizer.ttype==' ') // test for a space

    There are a lot of settings for StreamTokenizer. Which ones are you using.
    For example here are some from a program I have:
    Java Code:
             stzr.resetSyntax();             // Clear to allow digits as characters
    //         stzr.whitespaceChars(0, 32);   // this suppresses EOL
             stzr.whitespaceChars(0, 12);    // skip eol and nl >>THIS effects lineno()???
             stzr.whitespaceChars(14, 32);   
             stzr.wordChars('a', 'z');
             stzr.wordChars('A', 'Z');
             stzr.wordChars('/', '/');         // for /tags
             stzr.wordChars('\'','\'');        // for apostrophy???
             stzr.wordChars('0', '9');         // No numbers only text ??? DOES NOT WORK
             stzr.wordChars('.', '.');
             stzr.wordChars('-', '-');
             stzr.wordChars('#', '#');         // for Hex values
             stzr.quoteChar('"');
             stzr.ordinaryChar(' ');           // Return spaces
             stzr.eolIsSignificant(true);
    Last edited by Norm; 07-15-2010 at 08:20 PM.

  3. #3
    hedonist is offline Member
    Join Date
    Jun 2009
    Posts
    35
    Rep Power
    0

    Default

    Thank you Norm. I tried doing that but it doesn't work.

  4. #4
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,317
    Rep Power
    25

    Default

    but it doesn't work
    There is more than one setting to set. You need to show all that you've tried and how you coded them.

    Can you post a short executable program that demos the problem. Hardcode an input String and use a ByteArray stream or StringReader over it.

    For example:
    Java Code:
           String[] ss = {"<type> {name=\"Norm\"\naddr=\"Cal\"}\n",
                         "<type> {name=\"John\"\naddr=\"Penn\"}\n",
                         "<type> {name=Bill addr=Ohio} <type>{name=Ann addr=here}"};
    
          // Parse the strings
          for(int i=0; i < ss.length; i++) {
            System.out.println("<Parsed string>" + ss[i] + "<end of string>");
            try{
             StreamTokenizer theStream = new StreamTokenizer(new StringReader(ss[i]));

  5. #5
    hedonist is offline Member
    Join Date
    Jun 2009
    Posts
    35
    Rep Power
    0

    Default

    hi, here is my code:

    //tok represents instance of VB file
    //st represents StreamTokenizer

    if (tok.isLineBreak(st)) {
    //append '\n' to an external file so that codes in different lines are written in separate lines
    }

    public boolean isLineBreak(StreamTokenizer st) {
    return(st.ttype==';' || st.ttype=='{' ||st.ttype=='}' ||st.ttype==' ');
    }
    The purple color part doesn't work but for ';' and '{' it works.

  6. #6
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,317
    Rep Power
    25

    Default

    Can you post a short executable program that demos the problem
    I need a fully executable program, not just a few lines of code that are mostly comments.

    Try debugging the code by adding println()s to show the all the values you need to see.

  7. #7
    hedonist is offline Member
    Join Date
    Jun 2009
    Posts
    35
    Rep Power
    0

    Default

    Sorry about that. I am working with lots of files so its difficult to post the executable program. Thank you very much for the effort.

  8. #8
    Norm's Avatar
    Norm is offline Moderator
    Join Date
    Jun 2008
    Location
    SW Missouri
    Posts
    17,317
    Rep Power
    25

    Default

    I don't want your program. Can you make a small executable program that demos the problem? It would only have a sample input line, a StreamTokenizer and the testing of the parsed values. It shouldn't be over 100 lines total.

  9. #9
    hedonist is offline Member
    Join Date
    Jun 2009
    Posts
    35
    Rep Power
    0

    Default

    I had to use StreamTokenizer.resetSyntax() to make it work. Thank you Norm once again.

Similar Threads

  1. Trouble with Tokenizing String
    By ramathews in forum New To Java
    Replies: 0
    Last Post: 03-30-2010, 02:19 PM
  2. Replies: 3
    Last Post: 12-30-2009, 02:19 AM
  3. Convert java code to midlet code
    By coldvoice05 in forum Advanced Java
    Replies: 1
    Last Post: 08-09-2009, 01:21 PM
  4. String tokenizing with Scanner
    By vijaygk in forum Advanced Java
    Replies: 2
    Last Post: 07-15-2008, 04:44 AM
  5. Tokenizing with Scanner
    By sireesha in forum New To Java
    Replies: 3
    Last Post: 02-05-2008, 08:44 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •