Lucene Indexer Encoding problem

[svirid]

Good day guys,

hope u can help me. I am trying to index French and Russian documents with Lucene and have no luck. I am new in JAVA so basically I really need your help.

I was able to get text from pdfs, when I save it its all fine I can clearly see russian charachters in txt file but when I add it to the Index its all ??? or other garbage.

Here is what I do:

I first use PDF box to extract text.

textFile = "c:/java/faq.txt";
pdfFile = "c:/java/faq.pdf"; 

//FIRST I AM GETTING TEXT FROM PDF
document = PDDocument.load( pdfFile );

output = new OutputStreamWriter  ( new FileOutputStream  ( textFile ), "UTF-8" );	    
PDFTextStripper stripper = new PDFTextStripper();
stripper.setStartPage( 1 );
stripper.setEndPage( 20 );

//THIS SAVES TEXT INTO THE TXT FILE, TXT FILE COMPLETELY FINE
stripper.writeText(document, output);

//BUT WHEN I GET TEXT LIKE THAT TO ADD TO THE INDEX
textData = stripper.getText(document);


Analyzer analyzer = new StandardAnalyzer();        
Directory directory = FSDirectory.getDirectory("c:/java/collection");
IndexWriter iwriter = new IndexWriter(directory, analyzer, new IndexWriter.MaxFieldLength(250));
Document doc = new Document();
        
doc.add(new Field("fieldname", textData, Field.Store.YES, Field.Index.NOT_ANALYZED));
iwriter.addDocument(doc);
iwriter.optimize();
iwriter.close();

This code above properly saves extracted text to the txt file, whioch I dotn really need. What I want is to get text and add it to the Index right away. When I open index files in notepad I can see garbage instead of russian characters.

Please help. Thank you

[wolfcro]

Why don't you try to use the RussianAnalyzer (org.apache.lucene.analysis.ru.RussianAnalyzer) instead of the StandardAnalyzer which is made for english.

if you don't have the RussianAnalyzer already you can find it here:
svn.apache.org/repos/asf/lucene/java/trunk/contrib/

i haven't tried it myself, but i hope it will work for you

[svirid]

Thank you for your reply I will try to do that but I was kinda hopping that Lucene can handle it on its own. The problem is that when I am indexing a document I dont know what language its written with, russian, french or english.

[wolfcro]

lucene indexer uses an analyzer to recognize and extract the tokens in a text.

you can try different analyzers to see which works better for you.... there are many.... so I suggest to try them

if you don't know the exact language try to find an analyzer that works "good" with all..... but it will not be easy because of language specific encoding.....

you could also try to first recognize in a manner the language of the text and then use the apropriate analyzer
(try to look for specific characters ???)

[piyu.sha]

IF you try to read the index file you may not be able to relate it. Instead use Luke to browse the Lucene index. It can show ho the data is stored and you can also do some sample searches on same index.


---------------------
Music | HTS | Remix

[wolfcro]

we weren't talking about reading the index but creating it....
but for all there is appropriate api hehe :P