Chinese Word Segmentation Method Based on Reverse maximized Word Table ZZ

In the previous knowledge management system, because full-text search and Word Frequency Statistics need to be established, Chinese text must be segmented. For Chinese word segmentation,

It should be the Institute of Natural Sciences of the Chinese Emy of sciences, but it is relatively complicated. I did not understand it several times. :), because of our normal Knowledge System

The requirements for word segmentation are not so high, so we chose the most efficient word segmentation method. The word vocabulary is the 97 version of The People's Daily.

The actual effect can reach more than 90%, basically meeting the needs. Supports e.net word segmentation. The word table is loaded at one time at startup;

DetailsCodeAs follows:

Public sealed class ltwordtokenizer: tokenizer
{
Private string buffertext;
Private arraylist wordarray;
Private int intindex = 0;
Public static hashtable hsdic = new hashtable ();

Public ltwordtokenizer (textreader _ in)
{
Input = _ in;
Buffertext = input. readtoend (). tolower ();
Wordarray = new arraylist ();
Wordsegment (buffertext );

}

Public void wordsegment (string sentence)
{
Int senlen = sentence. length;
Int I = 0, j = 0;
Int M = 12;
String word;
While (I <senlen)
{
Int n = I + M <senlen? I + M: senlen + 1;
Bool bfind = false;
For (j = N-1; j> I; j --)
{
WORD = sentence. substring (I, j-I). Trim ();
If (hsdic. containskey (word. Trim ()))
{
Wordarray. Add (new token (word, I, I + word. Length ));
Bfind = true;
I = J;
Break;
}
}
If (! Bfind)
{
WORD = sentence. substring (I, 1). Trim ();
I = J + 1;
If (word. Trim ()! = "")
{
Wordarray. Add (new token (word, I, I + word. Length ));
}
}
}
}

 
Public override token next ()
{

If (intindex <wordarray. Count)
{
Intindex ++;
Return (token) (wordarray [intIndex-1]);
}
Else
Return NULL;
}

}
}

You can change it next time during word splitting, without having to save it to the arraylist for better dynamic speed.