index not reaching a value, still ArrayIndexOutOfBounds exception at that value

[Navin Israni]

ok..so I am doing a program on NLP..it uses function eliminateStopWords()..this function reads from a 2D array "sentTokens" (of detected tokens). In the code below, index i is sentence number, j is for each token in the ith sentence.

now, what my eliminateStopWords() does is this:
1. it reads stop words from a text file and stores them in a TreeSet
2. reads tokens from sentTokens array and checks them for stop words. If they are collocations, then they should not be checked for stop words, they are just dumped into a finalTokens array...if they are not a collcation, then they are individually checked for stop words and are added to finalTokens array only if they are not stop words..

the problem comes in the loop of this step 2..here is some code of it:

[COLOR="Green"]  /** This method is used to detects sentences */[/COLOR]

    private void detectSentences (String text){

                [COLOR="green"]/** SentenceDetection with OpenNLP API */
// InputStream --> SentenceModel --> SentenceDetectorME[/COLOR]

            try {
            modelIn=  new FileInputStream ("../OpenNLP/models/en-sent.bin");
            sentModel= new SentenceModel(modelIn);
            SentenceDetectorME sentenceDetector = new SentenceDetectorME(sentModel);
            dSentences = sentenceDetector.sentDetect(text);
            modelIn.close();
            NO_OF_SENTENCES = dSentences.length;
            System.out.println(NO_OF_SENTENCES);
            }
            catch (Exception e)
            {
              System.out.println(e);
            }

            [COLOR="green"]/** Display dSentences array, for testing purposes */[/COLOR]
            for (i=0; i < dSentences.length; i++)
                taFileContents.setText( taFileContents.getText()+ "\n Sentence " + (i+1) + " : " + dSentences[i]);

    }

[COLOR="green"]// This method tokenizes the sentences[/COLOR]
    private void iTokenize() {
        [COLOR="green"]// InputStream --> TokenizerModel --> TokenizerME[/COLOR]
        try {

            modelIn=  new FileInputStream ("../OpenNLP/models/en-token.bin");
            tokenModel = new TokenizerModel(modelIn);
            Tokenizer tokenizer = new TokenizerME(tokenModel);
            for(i=0;i < NO_OF_SENTENCES; i++)
                sentTokens[i] = tokenizer.tokenize( dSentences[i] );

            modelIn.close();
            finalTokens = new String [NO_OF_SENTENCES][];
            for(i=0; i < NO_OF_SENTENCES; i++)
               finalTokens[i]=new String [ sentTokens[i].length ];


        }
        catch(Exception e) {
            System.out.println(e);
        }

    }

private void eliminateStopWords() {
        try {
          [COLOR="green"]  // Loading TreeSet for stopwords from the file.[/COLOR]
            stopWords = new TreeSet<String> ();
            fin = new File("stopwords.txt");
            fScan = new Scanner(fin);
            while (fScan.hasNextLine()) 
                stopWords.add(fScan.nextLine());

            fScan.close();

            

            int k=0,l=0;    [COLOR="green"]// additional indices for finalTokens array[/COLOR]
            System.out.println(NO_OF_SENTENCES);

     newSentence: for(i=0; i < NO_OF_SENTENCES; i++)
             {
            System.out.println(i);    
                [COLOR="Red"]for (j=0; j < sentTokens[i].length - 1; j+=2)
                { // error is here
[/COLOR]
[COLOR="green"]// if current token is the last in the sentence, do not check for collocations, only check for stop words
// this is done to avoid ArrayIndexOutOfBounds Exception in sentences with odd number of tokens
[/COLOR]
                    if ( j == sentTokens[i].length - 1) {
                        String lastToken = sentTokens [i][j];
                        if (!stopWords.contains(lastToken))
                            finalTokens[k][l++] = lastToken;

                        [COLOR="green"]// after analyzing last token, move to analyzing the next sentence[/COLOR]

                        continue newSentence;

                    }
                   [COLOR="green"] // otherwsise, get two successive tokens[/COLOR]
                        String currToken = sentTokens[i][j];
                        String nextToken = sentTokens[i][j+1];

                        if ( isCollocation(currToken, nextToken) ) {

[COLOR="green"]// if the current and next tokens form a bigram collocation, they are not 
// checked for stop words but are directly dumped into finalTokens array
[/COLOR]
                            finalTokens[k][l++] = currToken;
                            finalTokens[k][l++] = nextToken;
                        }

                        if ( !stopWords.contains(currToken) )
                                finalTokens[k][l++] = currToken;

                        if ( !stopWords.contains(nextToken) )
                                finalTokens[k][l++] = nextToken;
                }

                k++;    [COLOR="green"]// next sentence in finalTokens array[/COLOR]
            }

           [COLOR="green"] // Test code to print finalTokens array[/COLOR]
               for(i=0; i < NO_OF_SENTENCES; i++) {
                   for (j=0; j < finalTokens[i].length; j++) 
                       System.out.print( finalTokens[i][j] + " " );
                  
                   System.out.println();
               }



        }
            catch (Exception e) {
                e.printStackTrace();
            }
    }

and here is the output and stack trace:

[COLOR="red"]11
0
java.lang.ArrayIndexOutOfBoundsException: 14 [/COLOR]
        at summarizer3.Summarizer.eliminateStopWords(Summarizer.java:294)
        at summarizer3.Summarizer.cmdSummarizeActionPerformed(Summarizer.java:607)
        at summarizer3.Summarizer.access$200(Summarizer.java:28)
        at summarizer3.Summarizer$3.actionPerformed(Summarizer.java:414)
        at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
        at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
        at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
        at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
        at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
        at java.awt.Component.processMouseEvent(Component.java:6289)
        at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
        at java.awt.Component.processEvent(Component.java:6054)
        at java.awt.Container.processEvent(Container.java:2041)
        at java.awt.Component.dispatchEventImpl(Component.java:4652)
        at java.awt.Container.dispatchEventImpl(Container.java:2099)
        at java.awt.Component.dispatchEvent(Component.java:4482)
        at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4577)
        at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
        at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
        at java.awt.Container.dispatchEventImpl(Container.java:2085)
        at java.awt.Window.dispatchEventImpl(Window.java:2478)
        at java.awt.Component.dispatchEvent(Component.java:4482)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:644)
        at java.awt.EventQueue.access$000(EventQueue.java:85)
        at java.awt.EventQueue$1.run(EventQueue.java:603)
        at java.awt.EventQueue$1.run(EventQueue.java:601)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
        at java.awt.EventQueue$2.run(EventQueue.java:617)
        at java.awt.EventQueue$2.run(EventQueue.java:615)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:614)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

see the output clearly...i have printed NO_OF_SENTENCES..its fine..i have also printed the outer index 'i' for each iteration..now, when it runs, it prints 'i' as 0 for first iteration and immediately throws ArrayIndexOutOfBounds exception:14

here is what i can make out from all this: i dont get where the value reaches 14 when i=0...basically, where the hell is the exception coming from? which array is going out of bounds?

UPDATE: now, i also printed the 'j' the index of inner loop...and again..even the innder loop throws AIOOBE in the first iteration itself..basically it cannot move beyond the first location in the sentTokens array...what to do??

[Junky]

Imagine sentTokens[i] has a length of 6 with valid indicies of 0 - 5.
Therefore sentTokens[i].length - 1 would have a value of 5.
Initial value of j is 0
You increment j by 2
2 is less than 5, keep looping
4 is less than 5, keep looping
j is now 6 and causes an AIOOBE

[Navin Israni]

hey thanks for reply, i kinda knew this but the thing is the statement where the exception is coming..yes, it is supposed to go AIOOBE but the error comes at the start of inner loop...even though j reaches to the value causing AIOOBE, the exception must be thrown only when i am actually accessing an AIOOB location...which i am not doing at the location where the error is actually occuring...(see red lines in the code)

and moreover, the AIOOBE is supposed to reach after multiple iterations...if previous iterations have been successful, it should print the indices that have been successfully iterated through..( i have printed 'i' before the red line where the error occurs)...but it only prints 0 i.e. the first iteration

now, i also printed the 'j' the index of inner loop..and even that prints only 0...and again..even the innder loop throws AIOOBE in the first iteration itself..basically it cannot move beyond the first location in the sentTokens array...what to do??

[Junky]

Copy and paste line 294

[Navin Israni]

 [COLOR="Red"]for (j=0; j < sentTokens[i].length - 1; j+=2)
                [/COLOR]

i think i have already colored it red in the code...