Sharing mysql full-text search: simple functions for Chinese Word Segmentation

Provides various official and user-released code examples. For code reference, you are welcome to share and learn. mysql full-text search: simple functions for Chinese Word Segmentation
Address: http://www.jb100.net/html/content-22-400-1.html
Some time ago, I studied Chinese full-text search and found that mysql does not support Chinese full-text search. However, there are some solutions: manually separate Chinese words with spaces and add in boolean mode to the search. However, Chinese Word Segmentation is a problem. This is a big problem. It seems that the Chinese Emy of Sciences has a group dedicated to Chinese word segmentation. If we use php for word segmentation, it is very difficult to implement true semantic word segmentation, even if the implementation efficiency is not high. Generally, we use the following method for Word Segmentation:

For example, we have a saying: Hello, I'm Liu chunlong.
So we can use this word segmentation: Hello, I am Liu Chunchun long.


This may seem silly, but it is feasible to use it in practice, because the keywords we enter during the search are also segmented by this method.

The following is a self-written function that can implement this word segmentation. Input three parameters:

1. the string to be segmented. Required: English, punctuation, numbers, Chinese characters, and Japanese. Code as UTF-8
2. Whether to return a string. Optional. The default value is. If false is input, an array is returned.
3. Whether to use base64_encode in Chinese. Optional. Default Value: Yes. Mysql full-text search has a configuration of ft_min_word_len. The value is generally 4, and the Chinese words we divide into are two words, which won't be considered a word by mysql. After base64_encode, the word length is 8, and there is no minimum length problem. Base64_encode increases the data size by 50%.


Note that both the input and output string encoding are UTF-8. Function string2words ($ s, $ return_string = true, $ encode64 = true)
{
$ Re = '';
// Match Chinese Characters
If (preg_match_all ("/([x {4e00}-x {9fff}] {2,})/u", $ s, $ ms ))
{
Foreach ($ ms [0] as $ w)
{
// Key part: Word Segmentation
$ L = strlen ($ w)/3;
For ($ I = 0; $ I <$ l; $ I ++)
{
$ Wi = substr ($ w, $ I * 3, 6 );
If (strlen ($ wi)> 3)
{
$ Re. = ($ encode64 )? '. Str_replace (', ',' @ ', base64_encode ($ wi):'. $ wi;
}
}
}
}
// Match numbers
If (preg_match_all ("/(d + [.]? D +)/", $ s, $ ms ))
{
Foreach ($ ms [0] as $ wi)
{
If (strlen ($ wi)> = 2)
{
$ Re. = ($ encode64 )? '. Str_replace (', ',' @ ', base64_encode ($ wi):'. $ wi;
}
}
$ S = preg_replace ("/(d + [.]? D +)/",'', $ s );
}
// Remove all double-byte characters
$ S = preg_replace ("/([^ x {00}-x {ff}] +)/u", '', $ s );
$ Re = $ s. ''. $ re;
If (! $ Return_string)
{
$ Re = preg_replace ("/([^ d]) ([,.-? N]) ([^ d])/", '$1 $ 3', $ re );
$ Re = trim (preg_replace ("/[s] {2,}/", '', $ re ));
$ Arr = explode ('', $ re );
$ Re = array ();
Foreach ($ arr as $)
{
If (strlen ($ a) >=2) $ re [] = $;
}
Return $ re;
}
Else
{
$ Re = trim (preg_replace ("/[s,.] {2,}/", '', $ re ));
Return $ re;
}
}Address: http://www.jb100.net/html/content-22-400-1.html

AD: truly free, domain name + VM + enterprise mailbox = 0 RMB