Lucene constructs synonym word breaker

in the more complex Lucene search business scenario, it is not enough to download a word breaker directly on the web as a project. So how to assess a Chinese word breaker's good and bad: Generally speaking, there are two points, thesaurus and search efficiency, which is the algorithm.

Lucene in the inverted table, different participle units have different Positionincrementattribute, if two words have the same positional attributes, For example: I define the United States and China these two words in the inverted list is the same position, then search the United States, China can also come out. This is the synonym search principle.

The following code (after using Mmseg's tokenizer to cut the word and then make synonyms):

Customize the word breaker first:

Package Hhc;import Java.io.reader;import Org.apache.lucene.analysis.analyzer;import Org.apache.lucene.analysis.tokenstream;import Com.chenlb.mmseg4j.dictionary;import Com.chenlb.mmseg4j.MaxWordSeg Import com.chenlb.mmseg4j.analysis.mmsegtokenizer;/** * Write a word breaker, you can generally refer to the original word breaker is how the wording of * @author HHC * */public class Mysameanalyzer extends analyzer{//synonyms private samewordcontext samewordcontext=null;public Mysameanalyzer ( Samewordcontext samewordcontext) {this.samewordcontext=samewordcontext;} @Overridepublic tokenstream Tokenstream (String fieldName, Reader Reader) {//Dictionary dic=dictionary.getinstance (); return new Mysametokenfilter (new Mmsegtokenizer (New Maxwordseg (DIC), reader), samewordcontext);}}

And then make a synonym for the tokenstream stream.

Package Hhc;import Java.io.ioexception;import Java.util.stack;import org.apache.lucene.analysis.tokenfilter;import Org.apache.lucene.analysis.tokenstream;import Org.apache.lucene.analysis.tokenattributes.CharTermAttribute; Import Org.apache.lucene.analysis.tokenattributes.positionincrementattribute;import Org.apache.lucene.util.attributesource;public class Mysametokenfilter extends Tokenfilter {//Word breaker Unit information private Chartermattribute CTA = null;//Position information Private Positionincrementattribute pia = null;//status Private attributesource.state Curr ent;//synonym Set private stack<string> sames = null;private Samewordcontext samewordcontext=null;protected Mysametokenfilter (Tokenstream input,samewordcontext samewordcontext) {super (input); CTA = Input.addattribute ( Chartermattribute.class);p ia = Input.addattribute (positionincrementattribute.class); Sames=new Stack<String> (); this.samewordcontext=samewordcontext;} @Overridepublic Boolean Incrementtoken () throws IOException {try {if (sames!=null&&sameS.size () > 0) {//delete the object on the stack, then return the function value on the object, and get this synonym string str = Sames.pop ();//Restore State restorestate (current); Cta.setempty ( ); Cta.append (str);p ia.setpositionincrement (0); return true;} If there is no data in the stream. if (!input.incrementtoken ()) return false;/** * Stream has data in it, make the corresponding synonym *///handle the information of the segmented word if (Existaddsameword (cta.tostring ())) {//hold current status first = Capturestate ();}} catch (Exception e) {//Todo:handle exceptione.printstacktrace ();} return true;} /** * Determine if the word breaker exists * * @param word * @return */private boolean existaddsameword (String word) {string[] Words=samewordc Ontext.getsameword (Word); if (words! = null) {for (String s:words) {Sames.push (s);} return true;} return false;}}

Lucene constructs synonym word breaker