Search engine (Chinese Word Segmentation)

[Disclaimer: All Rights Reserved. You are welcome to reprint it. Do not use it for commercial purposes. Contact Email: feixiaoxing @ 163.com]

Previously, we also talked about Chinese Word Segmentation when introducing search engines. Unlike English, all Chinese Characters in Chinese are connected, so one of our jobs is to split these words into phrases. Because only in this way can we build an index database and search for indexes, we can continue to build a search engine.

There are many word segmentation methods, such as left-to-right maximum length method, right-to-left maximum length method, least phrase method, Bayesian probability processing method, and so on. However, the key to the criteria of Word Segmentation lies in a good word segmentation dictionary. There are so many Chinese characters, but tens of thousands of Chinese characters are enough. However, if Chinese characters and Chinese characters are combined to form phrases, there will be more words, such as literary words, spoken words, names, place names, poems, and professional terms. Here, we can give you an example. We all like sogou's input method. On the one hand, its design is more user-friendly. On the other hand, it is not because it has many word libraries and is easy to use?

Some people think that word segmentation algorithms are very mysterious. In fact, writing a word segmentation algorithm is not complicated, so we can write a word segmentation algorithm with the maximum length traversal. Of course, only Chinese Word Segmentation is considered here. If it is English, numbers, traditional Chinese characters or symbols, it should be considered separately. This also verifies what we have repeatedly said before. It is not difficult to simply do one thing. The key is how to do well, do well, be efficient, and save costs.

 

# Include <stdio. h> # include <string. h> # include <memory. h> # include <malloc. h> # define length 256 static char * DIC [] = {"Beijing", "", "Peking University"}; # define number (sizeof (DIC) /sizeof (char *) Static char * buffer [length] = {0}; static int max_len = 0; static int min_len = 0; static int find_max_length () {int index; unsigned int Len; Len = 0; For (Index = 0; index <number; index ++) {If (LEN <strlen (DIC [Index]) {Len = strlen (DIC [Index]) ;}} return Len ;}static int find_min_length () {int index; unsigned int Len; Len = strlen (DIC [0]); for (Index = 1; index <number; index ++) {If (strlen (DIC [Index]) <Len) {Len = strlen (DIC [Index]);} return Len;} static void show_buffer (INT end) {int start; For (START = 0; Start <end; Start ++) {printf ("% s ", buffer [start]);} printf ("\ n");} static void _ process_segment (char * STR, int index) {int start; int Len; int found; char * data; char * label; If ('\ 0' = * Str) {show_buffer (INDEX); return;} label = STR + strlen (STR); retry: len = strlen (STR); If (LEN> length) {Len = length;} If (LEN> max_len) {Len = max_len;} found = 0; while (LEN> = min_len) {for (START = 0; Start <number; Start ++) {If (0 = strncmp (STR, DIC [start], Len )) {found = 1; break;} If (found) {break;} Len --;}/* if no Str was found, just step forward, but cannot beyond label */If (LEN <min_len & STR <label) {STR ++; goto retry;}/* if no Str was left, show all the Str */If (STR> = label) {show_buffer (INDEX); return;} DATA = (char *) malloc (LEN + 1 ); if (null = data) {return;} data [Len] = '\ 0'; memmove (data, STR, Len); buffer [Index] = data; _ process_segment (STR + Len, index + 1); free (data); buffer [Index] = NULL;} void process_segment (char * Str) {int length; if (null = Str) {return;} length = strlen (STR); If (length> length) {return;} _ process_segment (STR, 0 );} void segment_init () {min_len = find_min_length (); max_len = find_max_length (); memset (buffer, 0, sizeof (buffer);} int main (INT argc, char * argv []) {segment_init (); process_segment ("Peking University Japan"); return 1 ;}

 

The most complex function in the Code is the _ segment_process function, which involves a lot of situations. You can briefly introduce it:

(1) we recommend that you consider the minimum length and maximum length of the current dictionary during word segmentation to avoid unnecessary comparisons;

(2) If word segmentation fails to be searched, skip the next byte to continue searching;

(3) The function uses a recursive method. Pay attention to the function exit;

(4) Pay attention to floating processing of the buffer stack when writing functions;

(5) Pay attention to the basis and reason for determining the condition in the function.

(6) The number of DIC word lists is too small. To use process_segment for word segmentation, the number of word lists must be sufficient.