IK Word source Explanation (iv)-stop word processing __ algorithm and data structure

For the deactivation of words and the processing of the word not cut: Filter out the char_useless characters, including punctuation and unrecognized characters, pathmap stored in the Lexemepath set, find the adjacent lexemepath, the characters are not split between them verbatim character output, The original IK source code has Processunkowncjkchar (), in IK2012, this method was merged into the Outputtoresult (),

To deal with the ambiguity of participle

this. Arbitrator.process (context,this. Cfg.usesmart ());

Outputs the word segmentation results to the result set and processes the single CJK characters that are not sliced

Context.outputtoresult ();

Record the buffer displacement of this word

Context.markbufferoffset ();

/**

* Push Word segmentation results to the result set

* 1. Traverse from Buff head to this.cursor processed location

* 2. Push the result of the word segmentation in the map into results

* 3. CJDK characters that do not exist in the map are pushed into the results in a single word

*/

void Outputtoresult () {

int index = 0;

for (; index <=this. Cursor;) {

Skipping non-CJK characters

if (characterutil.char_useless == this. Chartypes[index]) {

index++;

continue;

}

Find the Lexemepath of the index position from Pathmap

Lexemepath Path = this. Pathmap.get (index);

if (path!=null) {

Lexeme to results collection in output Lexemepath

Lexeme L = Path.pollfirst ();

while (l!=null) {

this. Results.add (l);

After you move index to Lexeme

index= L.getbegin () + l.getlength ();

L= Path.pollfirst ();

if (l!=null) {

Output inside the path, words that are missing from the word

for (; index <l.getbegin (); index++) {

this. OUTPUTSINGLECJK (index);

}

}

}

The index corresponding Lexemepath is not found in theelse{//pathmap

Word output

this. OUTPUTSINGLECJK (index);

index++;

}

}

Empty the current map

this. pathmap.clear ();

}

There is a method in the word Code OUTPUTSTRINGCJK (index) is to output the word without segmentation.

Finally, take a look at the handling of discontinued words

/**

* Return to Lexeme

*

* Simultaneous processing of merging

* @return

*/

Lexeme Getnextlexeme () {

Remove from the result set, and remove the first lexme

Lexeme result = this. Results.pollfirst ();

while (Result!=null) {

Quantity Word Merging

this. Compound (result);

if (Dictionary.getsingleton (). Isstopword (this. Segmentbuff, Result.getbegin (), Result.getlength ())) {

Is the stop word continues to fetch the list next

result = this. Results.pollfirst ();

}Else{

Not stop word, generate lexeme word text, output

Result.setlexemetext (string.valueof (Segmentbuff, Result.getbegin (), Result.getlength ()));

break;

}

}

return result;

}

There is a isstopword in this method, which is used to determine whether the word is a deactivated word, and if it is deactivated, continue to take the next one in the list and process it. This process in the side of the word to determine whether to stop the word, you can debug to see more clearly.

Source Address: http://download.csdn.net/detail/a925907195/8240641