I have customized standard tokenizer for the reason we dont want to split the content by special chars, It should split only it found comma(,) or space.
But using same tokenizer i m facing problem with highlighter.
documents which received from searcher if sent to highlighter, its not highlighting any text.

below is the code:

TokenStream tokenStream = new MyAnalyzer(Lucene_version, stopWordsSet).tokenStream(fieldName, stringReader)
String result = highlighter.getBestFragments(tokenStream, no_of_fragments ,frag_sepeartor);

Does highlighter uses some different logic for tokenizer?


Appreciate your help.