Lucene constructs synonym word breaker

lucene4.0 version number has been used since tokenstreamcomponents instead of the Tokenstream stream. It contains filter and Tokenizer.

in the more complex Lucene search business scenario, it is not enough to download a word breaker directly on the web as a project. then how to assess a Chinese word breaker's good and bad: generally speaking. There are two points. Thesaurus and search efficiency, which is the algorithm.

Lucene the inverted permutation table, different participle units have different Positionincrementattribute, assuming that between two words Positionincrementattribute distance is 0. For Example: I define the United States and China these two words in the inverted list is the same position and distance of 0, then search the United States, China can also come out.

This is the synonym search principle.

The following code (after using Mmseg's tokenizer to cut words and then make synonyms):

Define your own word breaker first:

Package Hhc;import Java.io.reader;import Org.apache.lucene.analysis.analyzer;import Org.apache.lucene.analysis.tokenstream;import Com.chenlb.mmseg4j.dictionary;import Com.chenlb.mmseg4j.MaxWordSeg ; Import com.chenlb.mmseg4j.analysis.mmsegtokenizer;/** * Write a word breaker, generally can participate in the original word breaker is how the writing of * @author HHC * */public class Mysameanalyzer extends analyzer{//synonyms private samewordcontext samewordcontext=null;public Mysameanalyzer ( Samewordcontext samewordcontext) {this.samewordcontext=samewordcontext;} @Overridepublic tokenstream Tokenstream (String fieldName, Reader Reader) {//Dictionary dic=dictionary.getinstance (); return new Mysametokenfilter (new Mmsegtokenizer (New Maxwordseg (DIC), reader), samewordcontext);}}

And then make a synonym for the tokenstream stream.

Package Hhc;import Java.io.ioexception;import Java.util.stack;import org.apache.lucene.analysis.tokenfilter;import Org.apache.lucene.analysis.tokenstream;import Org.apache.lucene.analysis.tokenattributes.CharTermAttribute; Import Org.apache.lucene.analysis.tokenattributes.positionincrementattribute;import Org.apache.lucene.util.attributesource;public class Mysametokenfilter extends Tokenfilter {//Word breaker Unit information private Chartermattribute CTA = null;//Position information Private Positionincrementattribute pia = null;//status Private attributesource.state Curr ent;//synonym Set private stack<string> sames = null;private Samewordcontext samewordcontext=null;protected Mysametokenfilter (Tokenstream input,samewordcontext samewordcontext) {super (input); CTA = Input.addattribute ( Chartermattribute.class);p ia = Input.addattribute (positionincrementattribute.class); Sames=new Stack<String> (); this.samewordcontext=samewordcontext;} @Overridepublic Boolean Incrementtoken () throws IOException {try {if (sames!=null&&sameS.size () > 0) {//delete the object on the stack, and then return the function value on the object. And get this synonym for string str = Sames.pop ();//Restore Status restorestate (current); Cta.setempty (); cta.append (str); Pia.setpositionincrement (0); return true;} Suppose there is no data in the stream. if (!input.incrementtoken ()) return false;/** * Stream has data in it, make the corresponding synonym *///handle the information of the segmented word if (Existaddsameword (cta.tostring ())) {//hold current status first = Capturestate ();}} catch (Exception e) {//Todo:handle exceptione.printstacktrace ();} return true;} /** * Infer If the word breaker exists * * @param word * @return */private boolean existaddsameword (String word) {string[] Words=samewordc Ontext.getsameword (Word); if (words! = null) {for (String s:words) {Sames.push (s);} return true;} return false;}}

Lucene constructs synonym word breaker