Finally, it broke through the efficiency of Chinese word segmentation in cluence.

Traditional full-text retrieval is based on databases, and SQL Server Oracle MySQL provides full-text retrieval. However, these are relatively large and are not suitable for standalone or small applications. Program (MySQL or above can be used for integrated development), and MySQL does not support Chinese characters.
Later, I learned that Apache has an open-source full-text search engine, which is widely used. Lucene is a full-text search engine of Apache's Java version, with excellent performance. Unfortunately, it is a Java version, I have been thinking about whether there is C or C ++ version, and finally one day in http://sourceforge.net Amoy to a good stuff, clucene! Clucene is a full-text search engine for C ++ and is fully transplanted to Lucene. However, it does not support Chinese characters and has a lot of memory leaks.
Cluene does not support Chinese word segmentation, so I wrote a simple Chinese word segmentation. The general idea is the traditional binary word segmentation, because Chinese Word Segmentation is not similar to English languages, when we encounter space or punctuation, we think it is the end of a word. Therefore, we adopt the bipartite morphology, for example, Beijing, which is cut into Beijing and Beijing. In this way, the dictionary will be very large, but it is a simple word segmentation method (I will introduce some ideas about Chinese Word Segmentation later). Of course, you cannot enter "Beijing" during the search, so that you cannot retrieve it. If you enter "+ Beijing", you can retrieve Beijing. Although the accuracy is not very high, however, it is suitable for simple word segmentation and is not afraid to miss some words.
According to the clucene word segmentation module, I made a chinesetokenizer. This module is responsible for word segmentation. I wrote the main functions.

Chinesetokenizer. cpp:

Token * chinesetokenizer: Next (){

While (! Rd. Eos ())
{
Char_t CH = RD. getnext ();

If (isspace (char_t) CH )! = 0)
{
Continue;
}
// Read for alpha-nums and Chinese
If (isalnum (char_t) CH )! = 0)
{
Start = RD. Column ();

Return readchinese (CH );
}
}
Return NULL;
}

Token * chinesetokenizer: readchinese (const char_t PREV)
{
Bool ischinese = false;
Stringbuffer STR;
Str. append (prev );

Char_t CH = Prev;

If (char_t) CH> 8) & (char_t) CH> = 0xa0)
Ischinese = true;

While (! Rd. Eos () & isspace (char_t) CH) = 0)
{

Ch = RD. getnext ();

If (isalnum (char_t) CH )! = 0)
{< br> // enter the next space in mathematics or English. or the next Chinese Character
// is a Chinese character. read the next Chinese character to form a phrase, or read a space or an English ending.
If (ischinese)
{< br> // Chinese character, and ch is a Chinese character
If (char_t) CH> 8) & (char_t) CH> = 0xa0)
{< br> // returns the previous Chinese Character
Str. append (CH);
Rd. unget ();
// wprintf (_ T ("[% s]"), STR);
return new token (Str. getbuffer (), start, RD. column (), tokenimage [Lucene: Analysis: Chinese]);
}< br> else
{< br> // a letter, number, or space
Rd. unget ();
// wprintf (_ T ("[% s]"), STR);
return new token (Str. getbuffer (), start, RD. column (), tokenimage [Lucene: Analysis: Chinese]);
}< BR >}< br> else
{< br> // non-Chinese characters
/CH is a Chinese character
If (char_t) CH> 8) & (char_t) CH >=0xa0)
{< br> // wprintf (_ T ("[% s]"), STR );
Rd. unget ();
return new token (Str. getbuffer (), start, RD. column (), tokenimage [Lucene: Analysis: Chinese]);
}< br> Str. append (CH);
}< BR >}

}
// Wprintf (_ T ("[% s]"), STR );
Return new token (Str. getbuffer (), start, RD. Column (), tokenimage [Lucene: Analysis: Chinese: alphanum]);
}

At the same time, this Chinese word segmentation does not support files and can only support the form of memory streams, because I used RD. unget (); if it is a file, hey, I can only roll back half a byte.

Well. I wrote it here first. It's too urgent today. When I have time, I will send my clucene improvement to it.