Results 1 to 20 of 20
  1. #1
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default StringTokenizer versus Scanner concerning the delimiter

    The old fashioned Token class has the constructor StringTokenizer(...):
    StringTokenizer st = new StringTokenizer(data, "\t", true);
    Where if true "st" includes the delimiter.

    How could one get a similar result with the newer scanner class?

    Scanner sc = new Scanner(data);
    sc.useDelimiter("\t");
    But now "sc" does not include the tabs, but I want them to be there...
    So what to do?

  2. #2
    jim829 is offline Senior Member
    Join Date
    Jan 2013
    Location
    Northern Virginia, United States
    Posts
    3,423
    Rep Power
    5

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    In these situations I find the scanner class somewhat obtuse. IMHO, it doesn't process regexes in the hasNext() and next() methods as I think it should. I would suggest using the following to get all of your tokens including delimiters.

    Java Code:
    String testString = "To    be    or    not     to be     that is the question";
    Pattern testPattern = Pattern.compile("[ \t]|[^ \t]+");
    Matcher m = testPattern.matcher(testString);
    while (m.find()) {
       System.out.println("<" + m.group(0) + ">");
    }
    Regards,
    Jim
    The Java™ Tutorial | SSCCE | Java Naming Conventions
    Poor planning our your part does not constitute an emergency on my part.

  3. #3
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,863
    Rep Power
    19

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    Moved out of staff only area.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  4. #4
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    Here is the token code that works:

    Java Code:
                int tabGap = 8 * fm.stringWidth(" ");
                boolean isTab;	 // True when we find a Tab
    
                BufferedReader r = new BufferedReader(new FileReader(file));
                text.setText("Printing started");
                while((data = r.readLine()) != null) {
                    if (data.equals("")) {
                        data = " ";
                    }
                    StringTokenizer st = new StringTokenizer(data, "\t", true);
                    while (st.hasMoreElements() == true) { //read a line
                        String token = (String)st.nextElement();
                        isTab = token.equals("\t");
                        int stringWidth;
                        if (isTab == true) {
                            int nextCol = ((column + tabGap)/tabGap) * tabGap;
                            stringWidth = nextCol - column;
                        }
                        else {
                            stringWidth = fm.stringWidth(token);
                        }
    Here is the same code based on scanner:

    Java Code:
               int tabGap = 8 * fm.stringWidth(" ");
                boolean isTab;	 // True when we find a Tab
                BufferedReader r = new BufferedReader(new FileReader(file));
                text.setText("Printing started");
                while((data = r.readLine()) != null) {
                    if (data.equals("")) {
                        data = " ";
                    }
                    Scanner sc = new Scanner(data);
                    sc.useDelimiter("\t");
              //      StringTokenizer st = new StringTokenizer(data, "\t", true);
                    
                    while (sc.hasNext() == true) {
                      
                        String token = sc.next();
                        isTab = token.equals("\t"); // does not work because there are no tab present
                        int stringWidth;
                        if (isTab == true) {
                            int nextCol = ((column + tabGap)/tabGap) * tabGap;
                            stringWidth = nextCol - column;
                        }
                        else {
                            stringWidth = fm.stringWidth(token);
                        }
    So let see now where I will put your suggestion.
    Last edited by willemjav; 11-21-2013 at 10:42 AM.

  5. #5
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,863
    Rep Power
    19

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    I'm confused.
    You're splitting (and yes, I would use split rather than a Scanner) on a tab.
    So why not simply write the code such that every element bar the final one does the tab-related code?
    Java Code:
    for each String in the array
       handle data
       if not last element
          handle tab.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  6. #6
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    Can't get it going right, the inner if block doas the page printing.
    The question is where to put: g.drawString(token, column, line);


    Java Code:
        static void print() {
            // Print the file content
            text.setText("Printing file...");
            // Use the page and job attributes to
            // profile the print job...
            PrintJob pj = Toolkit.getDefaultToolkit().getPrintJob(
            frame, "Print File", job, page);
            if (pj == null) {
                text.setText("Printing cancelled");
                return;
            }
            int printerResolution = pj.getPageResolution();
            int topMargin = printerResolution;	 // 1 inch
            int leftMargin = printerResolution/2;	 // 0.5 inches
            Dimension pageSize = new Dimension(pj.getPageDimension());
            pageSize.width -= 2 *leftMargin;
            pageSize.height -= 2 * topMargin;
            // Use try/finally to release PrintJob even if we get an exception
            try {
                String data;
                // Set margin and clip
                Graphics g = pj.getGraphics();
                g.translate(leftMargin, topMargin);	// Skip margins
                g.clipRect(0, 0, pageSize.width, pageSize.height);
                g.setFont(font);
                // Get font metrics from the printer graphics
                fm = g.getFontMetrics();
                fontHeight = fm.getHeight();
                fontAscent = fm.getAscent();
                int line = fontAscent;	 // Pixel line on page
                int column = 0;	 // Pixel column on page
                int maxLine = pageSize.height - fontHeight;
                int tabGap = 8 * fm.stringWidth(" ");
                boolean isTab = false;	 // True when we find a Tab
                BufferedReader r = new BufferedReader(new FileReader(file));
                text.setText("Printing started");
                Pattern testPattern = Pattern.compile("\t");
                int stringWidth = 0;
                while((data = r.readLine()) != null) {
                    if (data.equals("")) {
                        data = " ";
                    }
                   
                    Matcher m = testPattern.matcher(data);
                    while (m.find()) {
                        isTab = m.equals("\t");
    
                        
                        if (isTab == true) {
                            int nextCol = ((column + tabGap)/tabGap) * tabGap;
                            stringWidth = nextCol - column;
                        }
                        else {
                            stringWidth = fm.stringWidth(data);
                        }
                    }
                    Scanner sc = new Scanner(data);
    
                    while (sc.hasNext() == true) {
                        String token = sc.next();
                        if (column + stringWidth > pageSize.width) {
                            line += fontHeight;
                            column = 0;
                            
                            if (line > maxLine) {
                                g.dispose();	 // Flush page
                                g = pj.getGraphics();	 // Access to printer
                                g.translate(leftMargin, topMargin);	// Skip margins
                                g.clipRect(0, 0, pageSize.width, pageSize.height);
                                g.drawString(token, column, line);
                                g.setFont(font);
                                line = fontAscent;
                                column = 0;
                            }
                            
                        }            
                        
                        column += stringWidth;
                    }
                    // End of line: go to next line
                    column = pageSize.width;
                }
    
                g.dispose();	 // Print the last page
            }
            catch (Exception e) {
                System.out.println("Exception while reading file data: " + e);
            }
            finally {
                // Always terminate the print job
                pj.end();
                text.setText("Printing completed");
            }
        }

  7. #7
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    ops, I did not see your last post, ok I will re-think it all....

  8. #8
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    So what's going wrong, here is the problem I am facing:
    -BufferedReader r represents all of the text file;
    -While((data = r.readLine()) != null), data is an actual line of the file;
    -StringWidth = fm.stringWidth(data), would measure the width of the line concerning the printer;
    -The stringWidth can not go over the pageSize.width.

    So the tabs are set to eight spaces:
    int tabGap = 8 * fm.stringWidth(" ");

    The tabs should be part of the line width to be calculated.
    The text that goes over the limit should go too the next line.
    Last edited by willemjav; 11-21-2013 at 12:04 PM.

  9. #9
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    So next the font is set, g.setFont(font), and the font heigth I get like this:
    fm = g.getFontMetrics();
    fontHeight = fm.getHeight();
    The way to fill the page would be (leaving a line open at page start and end):
    int maxLine = pageSize.height - (2*fontHeigh)t;
    After filling each line by asking: if (stringWidth < pageSize.width),
    counting each line like this: line += fontHeight;
    and preparing the page by doing: if ( line < maxLine)

    I theoretically could send the page to print and load a next one, right?

  10. #10
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    I began with a small while loop and noticed that the scanner
    doas not pick up tabs:

    Java Code:
    while((data = r.readLine()) != null) {
                    Scanner sc = new Scanner(data);
                  //  sc.useDelimiter("\t");
                   
                    while (sc.hasNext() == true) {
                        String token = sc.next();
                        
                        if (token.equals("\t")) {
                            System.out.print(" **tb** ");
                        }
                        else System.out.print(token + " ");
    
                    }
                     System.out.println("");
                }
    I have to think what to do with these stupid tabs!

    If there where no tabs I would take a string builder and add each token until
    reaching the printer's page size and give a new line.
    Scan is not the way to go I will look into split, because I need to read space and tabs!
    Last edited by willemjav; 11-21-2013 at 01:44 PM.

  11. #11
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,863
    Rep Power
    19

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    I explained what you need to do.
    Get an array of Strings (which is what String.split() does), based on the separator.
    Between each string do the tab stuff.

    That's it.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  12. #12
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    I got the tabs like this:

    Java Code:
    while((data = r.readLine()) != null) {
                    String token = "";
                    String[] parts = data.split("\t");
                    for (int x=0; x < parts.length; x++) {
                        token = token + parts[x] + "        "; // a tab of 8 spaces
                    }
                    System.out.println(token);
                }
    Still not there now comes the next part, measuring the string's length,
    so it'll fit on the page to print:
    if (fm.stringWidth(token) > pageSize.width)
    When the last added token passes the limit,
    I need to take it of and warp to the next line....

  13. #13
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,863
    Rep Power
    19

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    Remember, you won't want the tab spacing for the last element.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  14. #14
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    almost there, any comments:

    Java Code:
    String newlineToken = "";
                while((data = r.readLine()) != null) {
                    String token = newlineToken;
                    String tempToken = "";
                    String[] parts = data.split("\t");
                    for (int x=0; x < parts.length; x++) {
                        tempToken = token + parts[x] + "        "; // 8 spaces added as tab
                        if (fm.stringWidth(tempToken) > pageSize.width) {
                            newlineToken = parts[x];
                            break;
                        }
                        else {
                            token = tempToken;
                            newlineToken = "";
                        }
                    }
                    System.out.println(token);
                   
    
                }
    Last edited by willemjav; 11-21-2013 at 03:58 PM.

  15. #15
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    "Remember, you won't want the tab spacing for the last element."

    What you mean by that, tolls?
    Tabs might come where ever in the text, right?

    But anyway the code is still not working because,
    what if the token-string goes over the line-limit without a tab!
    Therefor I might need to go word by word using space as a split,
    or even character by character!
    And than in general: what about the tabs?
    Last edited by willemjav; 11-21-2013 at 04:03 PM.

  16. #16
    Tolls is offline Moderator
    Join Date
    Apr 2009
    Posts
    11,863
    Rep Power
    19

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    Take the following line:
    something\tsomething else\tanother thing

    When split on '\t' (a tab) you will get an array of three elements:
    [something,something else,another thing]

    Your routine above will then create an output with an additional 8 space "tab" at the end, which was not there in the original.
    Please do not ask for code as refusal often offends.

    ** This space for rent **

  17. #17
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    The original code with the StringTokenizer worked out fine so far:


    Java Code:
    while((data = r.readLine()) != null) {
                    if (data.equals("")) {
                    data = " ";
                    }
                    StringTokenizer st = new StringTokenizer(data, "\t", true);
                    while (st.hasMoreElements() == true) {
                        String token = (String)st.nextElement();
                        isTab = token.equals("\t");
                        int stringWidth;
                        if (isTab == true) {
                            int nextCol = ((column + tabGap)/tabGap) * tabGap;
                            stringWidth = nextCol - column;
                        } else {
                            stringWidth = fm.stringWidth(token);
                        }
                        if (column + stringWidth > pageSize.width) {
                                line += fontHeight;
                                column = 0;
                            if (isTab == true) {
                                stringWidth = tabGap;
                            }
                            if (line > maxLine) {
                                g.dispose();	 // Flush page
                                g = pj.getGraphics();	 // Access to printer
                                g.translate(leftMargin, topMargin);	// Skip margins
                                g.clipRect(0, 0, pageSize.width, pageSize.height);
                                g.setFont(font);
                                line = fontAscent;
                                column = 0;
                            }
                        }
                        if (isTab == false) {
                        g.drawString(token, column, line);
                        }
                        column += stringWidth;
                    }
                    // End of line: go to next line
                    column = pageSize.width;
                }

  18. #18
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    I see what you're saying!

  19. #19
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    Not very elegant but this will resolve the added tab:

    Java Code:
     while((data = r.readLine()) != null) {
                    String token = newlineToken;
                    String tempToken = "";
                    String tabgab = "        ";
                    String[] parts = data.split("\t");
                    for (int x=0; x < parts.length; x++) {
                        if (x == parts.length-1) tabgab = "";
                        tempToken = token + parts[x] + tabgab;
                        if (fm.stringWidth(tempToken) > pageSize.width) {
                            newlineToken = parts[x];
                            break;
                        }
                        else {
                            token = tempToken;
                            newlineToken = "";
                        }
                    }
                    System.out.println(token);
                   
    
                }
    Last edited by willemjav; 11-21-2013 at 05:01 PM.

  20. #20
    willemjav is offline Senior Member
    Join Date
    Dec 2007
    Location
    Spain
    Posts
    1,075
    Rep Power
    8

    Default Re: StringTokenizer versus Scanner concerning the delimiter

    This is close but still not correct because in case of a break,
    the remaining array should go to the next line not only parts[x]!
    Have to take paper and pencil and begin from scratch....

Similar Threads

  1. Scanner Delimiter
    By Games2Design in forum New To Java
    Replies: 1
    Last Post: 03-22-2013, 05:16 PM
  2. 32-bit versus 64-bit
    By newbie123 in forum New To Java
    Replies: 4
    Last Post: 09-06-2012, 05:52 PM
  3. Scanner delimiter issue
    By PrinceSendai in forum New To Java
    Replies: 3
    Last Post: 10-05-2010, 09:41 AM
  4. Replies: 1
    Last Post: 02-20-2009, 02:06 PM
  5. Use of Scanner class and Delimiter
    By tjhodge in forum New To Java
    Replies: 3
    Last Post: 02-12-2009, 05:26 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •