Results 1 to 5 of 5
  1. #1
    Navin Israni is offline Member
    Join Date
    Feb 2011
    Posts
    11
    Rep Power
    0

    Default index not reaching a value, still ArrayIndexOutOfBounds exception at that value

    ok..so I am doing a program on NLP..it uses function eliminateStopWords()..this function reads from a 2D array "sentTokens" (of detected tokens). In the code below, index i is sentence number, j is for each token in the ith sentence.

    now, what my eliminateStopWords() does is this:
    1. it reads stop words from a text file and stores them in a TreeSet
    2. reads tokens from sentTokens array and checks them for stop words. If they are collocations, then they should not be checked for stop words, they are just dumped into a finalTokens array...if they are not a collcation, then they are individually checked for stop words and are added to finalTokens array only if they are not stop words..

    the problem comes in the loop of this step 2..here is some code of it:

    Java Code:
    [COLOR="Green"]  /** This method is used to detects sentences */[/COLOR]
    
        private void detectSentences (String text){
    
                    [COLOR="green"]/** SentenceDetection with OpenNLP API */
    // InputStream --> SentenceModel --> SentenceDetectorME[/COLOR]
    
                try {
                modelIn=  new FileInputStream ("../OpenNLP/models/en-sent.bin");
                sentModel= new SentenceModel(modelIn);
                SentenceDetectorME sentenceDetector = new SentenceDetectorME(sentModel);
                dSentences = sentenceDetector.sentDetect(text);
                modelIn.close();
                NO_OF_SENTENCES = dSentences.length;
                System.out.println(NO_OF_SENTENCES);
                }
                catch (Exception e)
                {
                  System.out.println(e);
                }
    
                [COLOR="green"]/** Display dSentences array, for testing purposes */[/COLOR]
                for (i=0; i < dSentences.length; i++)
                    taFileContents.setText( taFileContents.getText()+ "\n Sentence " + (i+1) + " : " + dSentences[i]);
    
        }
    
    [COLOR="green"]// This method tokenizes the sentences[/COLOR]
        private void iTokenize() {
            [COLOR="green"]// InputStream --> TokenizerModel --> TokenizerME[/COLOR]
            try {
    
                modelIn=  new FileInputStream ("../OpenNLP/models/en-token.bin");
                tokenModel = new TokenizerModel(modelIn);
                Tokenizer tokenizer = new TokenizerME(tokenModel);
                for(i=0;i < NO_OF_SENTENCES; i++)
                    sentTokens[i] = tokenizer.tokenize( dSentences[i] );
    
                modelIn.close();
                finalTokens = new String [NO_OF_SENTENCES][];
                for(i=0; i < NO_OF_SENTENCES; i++)
                   finalTokens[i]=new String [ sentTokens[i].length ];
    
    
            }
            catch(Exception e) {
                System.out.println(e);
            }
    
        }
    
    private void eliminateStopWords() {
            try {
              [COLOR="green"]  // Loading TreeSet for stopwords from the file.[/COLOR]
                stopWords = new TreeSet<String> ();
                fin = new File("stopwords.txt");
                fScan = new Scanner(fin);
                while (fScan.hasNextLine()) 
                    stopWords.add(fScan.nextLine());
    
                fScan.close();
    
                
    
                int k=0,l=0;    [COLOR="green"]// additional indices for finalTokens array[/COLOR]
                System.out.println(NO_OF_SENTENCES);
    
         newSentence: for(i=0; i < NO_OF_SENTENCES; i++)
                 {
                System.out.println(i);    
                    [COLOR="Red"]for (j=0; j < sentTokens[i].length - 1; j+=2)
                    { // error is here
    [/COLOR]
    [COLOR="green"]// if current token is the last in the sentence, do not check for collocations, only check for stop words
    // this is done to avoid ArrayIndexOutOfBounds Exception in sentences with odd number of tokens
    [/COLOR]
                        if ( j == sentTokens[i].length - 1) {
                            String lastToken = sentTokens [i][j];
                            if (!stopWords.contains(lastToken))
                                finalTokens[k][l++] = lastToken;
    
                            [COLOR="green"]// after analyzing last token, move to analyzing the next sentence[/COLOR]
    
                            continue newSentence;
    
                        }
                       [COLOR="green"] // otherwsise, get two successive tokens[/COLOR]
                            String currToken = sentTokens[i][j];
                            String nextToken = sentTokens[i][j+1];
    
                            if ( isCollocation(currToken, nextToken) ) {
    
    [COLOR="green"]// if the current and next tokens form a bigram collocation, they are not 
    // checked for stop words but are directly dumped into finalTokens array
    [/COLOR]
                                finalTokens[k][l++] = currToken;
                                finalTokens[k][l++] = nextToken;
                            }
    
                            if ( !stopWords.contains(currToken) )
                                    finalTokens[k][l++] = currToken;
    
                            if ( !stopWords.contains(nextToken) )
                                    finalTokens[k][l++] = nextToken;
                    }
    
                    k++;    [COLOR="green"]// next sentence in finalTokens array[/COLOR]
                }
    
               [COLOR="green"] // Test code to print finalTokens array[/COLOR]
                   for(i=0; i < NO_OF_SENTENCES; i++) {
                       for (j=0; j < finalTokens[i].length; j++) 
                           System.out.print( finalTokens[i][j] + " " );
                      
                       System.out.println();
                   }
    
    
    
            }
                catch (Exception e) {
                    e.printStackTrace();
                }
        }
    and here is the output and stack trace:
    Java Code:
    [COLOR="red"]11
    0
    java.lang.ArrayIndexOutOfBoundsException: 14 [/COLOR]
            at summarizer3.Summarizer.eliminateStopWords(Summarizer.java:294)
            at summarizer3.Summarizer.cmdSummarizeActionPerformed(Summarizer.java:607)
            at summarizer3.Summarizer.access$200(Summarizer.java:28)
            at summarizer3.Summarizer$3.actionPerformed(Summarizer.java:414)
            at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
            at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
            at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
            at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
            at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
            at java.awt.Component.processMouseEvent(Component.java:6289)
            at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
            at java.awt.Component.processEvent(Component.java:6054)
            at java.awt.Container.processEvent(Container.java:2041)
            at java.awt.Component.dispatchEventImpl(Component.java:4652)
            at java.awt.Container.dispatchEventImpl(Container.java:2099)
            at java.awt.Component.dispatchEvent(Component.java:4482)
            at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4577)
            at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
            at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
            at java.awt.Container.dispatchEventImpl(Container.java:2085)
            at java.awt.Window.dispatchEventImpl(Window.java:2478)
            at java.awt.Component.dispatchEvent(Component.java:4482)
            at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:644)
            at java.awt.EventQueue.access$000(EventQueue.java:85)
            at java.awt.EventQueue$1.run(EventQueue.java:603)
            at java.awt.EventQueue$1.run(EventQueue.java:601)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
            at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
            at java.awt.EventQueue$2.run(EventQueue.java:617)
            at java.awt.EventQueue$2.run(EventQueue.java:615)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
            at java.awt.EventQueue.dispatchEvent(EventQueue.java:614)
            at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
            at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
            at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
            at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
            at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
            at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
    see the output clearly...i have printed NO_OF_SENTENCES..its fine..i have also printed the outer index 'i' for each iteration..now, when it runs, it prints 'i' as 0 for first iteration and immediately throws ArrayIndexOutOfBounds exception:14

    here is what i can make out from all this: i dont get where the value reaches 14 when i=0...basically, where the hell is the exception coming from? which array is going out of bounds?

    UPDATE: now, i also printed the 'j' the index of inner loop...and again..even the innder loop throws AIOOBE in the first iteration itself..basically it cannot move beyond the first location in the sentTokens array...what to do??
    Last edited by Navin Israni; 04-14-2011 at 08:20 AM.

  2. #2
    Junky's Avatar
    Junky is offline Grand Poobah
    Join Date
    Jan 2011
    Location
    Dystopia
    Posts
    3,800
    Rep Power
    7

    Default

    Imagine sentTokens[i] has a length of 6 with valid indicies of 0 - 5.
    Therefore sentTokens[i].length - 1 would have a value of 5.
    Initial value of j is 0
    You increment j by 2
    2 is less than 5, keep looping
    4 is less than 5, keep looping
    j is now 6 and causes an AIOOBE

  3. #3
    Navin Israni is offline Member
    Join Date
    Feb 2011
    Posts
    11
    Rep Power
    0

    Default

    hey thanks for reply, i kinda knew this but the thing is the statement where the exception is coming..yes, it is supposed to go AIOOBE but the error comes at the start of inner loop...even though j reaches to the value causing AIOOBE, the exception must be thrown only when i am actually accessing an AIOOB location...which i am not doing at the location where the error is actually occuring...(see red lines in the code)

    and moreover, the AIOOBE is supposed to reach after multiple iterations...if previous iterations have been successful, it should print the indices that have been successfully iterated through..( i have printed 'i' before the red line where the error occurs)...but it only prints 0 i.e. the first iteration

    now, i also printed the 'j' the index of inner loop..and even that prints only 0...and again..even the innder loop throws AIOOBE in the first iteration itself..basically it cannot move beyond the first location in the sentTokens array...what to do??

  4. #4
    Junky's Avatar
    Junky is offline Grand Poobah
    Join Date
    Jan 2011
    Location
    Dystopia
    Posts
    3,800
    Rep Power
    7

    Default

    Copy and paste line 294

  5. #5
    Navin Israni is offline Member
    Join Date
    Feb 2011
    Posts
    11
    Rep Power
    0

    Default

    Java Code:
     [COLOR="Red"]for (j=0; j < sentTokens[i].length - 1; j+=2)
                    [/COLOR]

    i think i have already colored it red in the code...

Similar Threads

  1. ArrayIndexOutofBounds Exception on an Excel File
    By CrashWave in forum New To Java
    Replies: 3
    Last Post: 03-01-2011, 09:44 PM
  2. Help Array Index out of bounds exception
    By star400040 in forum New To Java
    Replies: 2
    Last Post: 12-10-2010, 11:24 PM
  3. Array index Out of bound Exception
    By nitin_daviet88 in forum New To Java
    Replies: 9
    Last Post: 07-28-2010, 06:32 AM
  4. ArrayIndexOutofBounds Exception
    By atul.goldenstring in forum New To Java
    Replies: 10
    Last Post: 04-10-2010, 11:47 AM
  5. Array Index Out of Bounds Exception
    By kool001 in forum New To Java
    Replies: 1
    Last Post: 12-03-2009, 08:42 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •