Results 1 to 1 of 1
  1. #1
    gadek.a is offline Member
    Join Date
    Jan 2011
    Posts
    3
    Rep Power
    0

    Default adding field with tokenStream Field(name, tokenStream, termVector) constructor

    I have to deal with serious bug problem. My program doesn't work correctly. Problem shows probably in this part of code:

    Java Code:
    StandardAnalyzer stAnalyzer = new StandardAnalyzer(Version.LUCENE_31);
    TokenStream stStream = stAnalyzer.tokenStream("analizedContent", new StringReader(handler.toString()));
    SimpleAnalyzer siAnalyzer = new SimpleAnalyzer(Version.LUCENE_31);
    TokenStream siStream = siAnalyzer.tokenStream("content", new StringReader(handler.toString()));
    		    	
    TermAttribute term = siStream.addAttribute(TermAttribute.class);
    while(siStream.incrementToken()){
          System.out.print(term.term() + ":");
    }
    System.out.println();
    doc.add(new Field("analizedContent", stStream, TermVector.YES));   // 10 NO
    doc.add(new Field("content", siStream, TermVector.YES));   // 10 NO
    
    System.out.println("en  string Value  " + doc.getField("analizedContent").stringValue());	
    System.out.println("en  to String  " + doc.getField("analizedContent").toString());
    System.out.println("en  isStored  " + doc.getField("analizedContent").isStored());
    I get some text from TIKA, and put it into two different analyzers. after that I'm adding two fields into doc. After all when I'am checking content of those fields (like in this code or later in the lucene index they are empty.
    It seems that both fields after all have "null" value. Any one can help? I'm out of time to make further research.

    Andrew



    update ->

    I have find out that the problem comes from those constructors:

    new Field(name, tokenStream)
    new Field(name, tokenStream, termVector)

    how?

    Java Code:
    StandardAnalyzer stAnalyzer = new StandardAnalyzer(Version.LUCENE_30);
    TokenStream stStream = stAnalyzer.tokenStream("analizedContent", new StringReader(handler.toString()));
     
    TermAttribute term = stStream.addAttribute(TermAttribute.class);
    System.out.println("loop 1");
    while(stStream.incrementToken()){
    System.out.print(term.term() + ":");
    }
    System.out.println();
    stStream.reset();
    
    Field field = new Field("analizedContent", stStream, TermVector.YES);
    field.setTokenStream(stStream);
    
    TokenStream tSV = field.tokenStreamValue();
    TermAttribute term2 = stStream.addAttribute(TermAttribute.class);
    
    System.out.println("loop 2");
    while(tSV.incrementToken()){
    System.out.print(term2.term() + ":");
    }
    System.out.println();
    
    System.out.println ("field.readerValue(): " + field.toString());
    System.out.println ("field.readerValue(): " + field.readerValue());
    System.out.println ("fieldfield.stringValue(): " + field.stringValue());
    and now what a get in console:
    [example]

    loop 1
    welcome:q&a:professional:enthusiast:programmers:ch eck:out:faq:stack:exchange:log:careers:chat:meta:a bout:faq:stack:overflow:questions:tags:users:badge s:unanswered:ask:question:what:difference:between: getpath:getabsolutepath:getca... /*and many more ;-)*/

    field.readerValue(): indexed,tokenized,termVector<analizedContent:>
    loop 2

    field.readerValue(): null
    fieldfield.stringValue(): null
    [/example]

    my comment: after creation of new Field we lose a value of posted tokenStream. So problem o curse when U want to index pre-analized text, for example with different then default analyzer (in my case polish one)

    my conclusion: something is not working wright!
    Probably nobody will help me so i'm going to find some alternate way to do this. For example will make String form analyzed text and then use default analyzer to parse it.


    Andrew
    Last edited by gadek.a; 07-24-2011 at 06:15 PM.

Similar Threads

  1. front end display of field description when clicking the field name
    By neils in forum JavaServer Pages (JSP) and JSTL
    Replies: 0
    Last Post: 10-29-2010, 12:47 PM
  2. Replies: 1
    Last Post: 10-25-2010, 01:20 PM
  3. Replies: 4
    Last Post: 01-18-2010, 12:13 AM
  4. Constructor and instance field
    By MIA6 in forum New To Java
    Replies: 7
    Last Post: 11-02-2009, 07:44 PM
  5. Adding Hyperlinks/Links inside a field of a JTable
    By javanewbie in forum AWT / Swing
    Replies: 3
    Last Post: 05-31-2009, 05:19 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •