Meng new note--c++ string strings (utf-8 Encoding) into a single word (can be mixed in english)

Recently in the dictionary under construction, using the Trie dictionary tree, you need to break the string into a single Word. Because the strings that are passed in May contain Chinese or english, they are not the same number of Bytes. At first naïve to think that Chinese is two bytes, so very happy directly to determine the current position of the character ASCII code is in 0~127, if it is to extract a character, otherwise extract two. There is a problem with this approach when testing the word-splitting Effect. For example, I pass a "abcde 12345" in, ABCDE can normally be decomposed into a B c D e, and the back of "12345" is Garbled.

So I started a trip to google, "how to break down a string in C + + in the Chinese into a single word" and so on, most of the methods of search and my previous methods, the code copy down the direct run is also garbled. It occurred to me that one of the reasons that Linux might appear garbled in Chinese was the coding problem, so I opened the vim configuration file and found that I really set the Chinese to utf-8.

After discovering this, i specifically searched for utf-8, and learned that it was a variable length code, with the following specific rules:

1) for a single-byte symbol, the first bit of the byte is set to 0, and the next 7 bits are the Unicode code for the Symbol. So for the English alphabet, the UTF-8 encoding and ASCII code are the Same.

2) for n-byte notation (n>1), the first n bits are set to 1, the N+1 bit is set to 0, and the first two bits of the subsequent bytes are set to 10. The rest of the bits are not mentioned, all of which are Unicode codes for this Symbol.

As Table:

1 bytes	0xxxxxxx
2 bytes	110xxxxx 10xxxxxx
3 bytes	1110xxxx 10xxxxxx 10xxxxxx
4 bytes	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
5 bytes	111110XX 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6 bytes	1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

With this, the idea is clear: first, I want to judge after a word is a few bytes, and then intercept the corresponding number of Bytes. So we have the following code:

1 voidDictionary::splitword (Const string& word, vector<string> &Characters)2 {3     intnum =word.size ();4     inti =0;5      while(i <Num)6     {7         intsize; 8         if(word[i] &0x80)9         {Ten             if(word[i] &0x20) one             { a                 if(word[i] &0x10) -                 { -                     if(word[i] &0x08) the                     { -                         if(word[i] &0x04) -                         { -Size =6; +}Else{ -Size =5; +                         } a}Else{ atSize =4; -                     } -}Else{ -Size =3; -                 } -}Else{ inSize =2; -             } to}Else{ +Size =1; -         } the         stringsubword; *Subword =word.substr (i, size); $ Characters.push_back (subword);Panax Notoginsengi + =size; -     } the}

If nested if, Although the process is clear, but the number of lines of code is too many, so modify it, get the following code:

1 voidDictionary::splitword (Const string& word, vector<string> &Characters)2 {3     intnum =word.size ();4     inti =0;5      while(i <Num)6     {7         intSize =1;8         if(word[i] &0x80)9         {Ten             Chartemp =word[i]; oneTemp <<=1; a              do{ -Temp <<=1; -++size; the} while(temp &0x80); -         } -         stringsubword; -Subword =word.substr (i, size); + Characters.push_back (subword); -i + =size; +     } a}

Less than half a year or so.

The result of decomposition is the existence of vector containers, which can be changed according to the specific NEEDS.

Finally found that the Chinese in the Utf-8 encoding is three bytes

In fact, only need to manually print out the corresponding string size, you can calculate how many bytes each word, then how did not find it?

Meng new note--c++ string strings (utf-8 Encoding) into a single word (can be mixed in english)