adding field with tokenStream Field(name, tokenStream, termVector) constructor

[gadek.a]

I have to deal with serious bug problem. My program doesn't work correctly. Problem shows probably in this part of code:

StandardAnalyzer stAnalyzer = new StandardAnalyzer(Version.LUCENE_31);
TokenStream stStream = stAnalyzer.tokenStream("analizedContent", new StringReader(handler.toString()));
SimpleAnalyzer siAnalyzer = new SimpleAnalyzer(Version.LUCENE_31);
TokenStream siStream = siAnalyzer.tokenStream("content", new StringReader(handler.toString()));
		    	
TermAttribute term = siStream.addAttribute(TermAttribute.class);
while(siStream.incrementToken()){
      System.out.print(term.term() + ":");
}
System.out.println();
doc.add(new Field("analizedContent", stStream, TermVector.YES));   // 10 NO
doc.add(new Field("content", siStream, TermVector.YES));   // 10 NO

System.out.println("en  string Value  " + doc.getField("analizedContent").stringValue());	
System.out.println("en  to String  " + doc.getField("analizedContent").toString());
System.out.println("en  isStored  " + doc.getField("analizedContent").isStored());

I get some text from TIKA, and put it into two different analyzers. after that I'm adding two fields into doc. After all when I'am checking content of those fields (like in this code or later in the lucene index they are empty.
It seems that both fields after all have "null" value. Any one can help? I'm out of time to make further research.

Andrew



update ->

I have find out that the problem comes from those constructors:

new Field(name, tokenStream)
new Field(name, tokenStream, termVector)

how?

StandardAnalyzer stAnalyzer = new StandardAnalyzer(Version.LUCENE_30);
TokenStream stStream = stAnalyzer.tokenStream("analizedContent", new StringReader(handler.toString()));
 
TermAttribute term = stStream.addAttribute(TermAttribute.class);
System.out.println("loop 1");
while(stStream.incrementToken()){
System.out.print(term.term() + ":");
}
System.out.println();
stStream.reset();

Field field = new Field("analizedContent", stStream, TermVector.YES);
field.setTokenStream(stStream);

TokenStream tSV = field.tokenStreamValue();
TermAttribute term2 = stStream.addAttribute(TermAttribute.class);

System.out.println("loop 2");
while(tSV.incrementToken()){
System.out.print(term2.term() + ":");
}
System.out.println();

System.out.println ("field.readerValue(): " + field.toString());
System.out.println ("field.readerValue(): " + field.readerValue());
System.out.println ("fieldfield.stringValue(): " + field.stringValue());

and now what a get in console:
[example]

loop 1
welcome:q&a:professional:enthusiast:programmers:ch eck:out:faq:stack:exchange:log:careers:chat:meta:a bout:faq:stack:overflow:questions:tags:users:badge s:unanswered:ask:question:what:difference:between: getpath:getabsolutepath:getca... /*and many more ;-)*/

field.readerValue(): indexed,tokenized,termVector<analizedContent:>
loop 2

field.readerValue(): null
fieldfield.stringValue(): null
[/example]

my comment: after creation of new Field we lose a value of posted tokenStream. So problem o curse when U want to index pre-analized text, for example with different then default analyzer (in my case polish one)

my conclusion: something is not working wright!
Probably nobody will help me so i'm going to find some alternate way to do this. For example will make String form analyzed text and then use default analyzer to parse it.


Andrew