C + + to determine if the string is UTF8 encoded

UTF8 is the most basic unit of 8bits or 1Bytes encoding, of course, it can also be based on 16bits and 32bits, respectively, called UTF16 and UTF32, but the current use is not much, and UTF8 is widely used in file storage and network transmission.

Coding principle

Look at this template first:

UCS-4 range (hex.) UTF-8 octet sequence (binary)

0000 0000-0000 007F 0xxxxxxx

0000 0080-0000 07FF 110xxxxx 10xxxxxx

0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx

0001 0000-001f FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

0020 0000-03ff FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

0400 0000-7fff FFFF 1111110x 10xxxxxx ... 10xxxxxx

Encoding steps:

1) First determine how many 8bits (octets) are required

2) Fill each octets's high bits with the above template

3) Fill the character bits into X, character order: Low → High, UTF8 order: Last octet lowest x→ first octet highest bit X

According to UTF8 encoding, up to 6 bytes, so UTF8 is a 1-6-byte encoding composition

int IsTextUTF8 (char* str,ulonglong length) {int I;dword nbytes=0;//uft8 can be encoded with 1-6 bytes, ASCII with one byte Uchar chr; BOOL ballascii=true; If all is ASCII, the description is not utf-8for (i=0;i<length;i++) {chr= * (str+i);//Determine if ASCII is encoded, if not, it may be utf-8,ascii with 7-bit encoding, but with one byte to save , the highest bit mark is 0,o0xxxxxxxif ((chr&0x80)! = 0) ballascii= false;if (nbytes==0)//If it is not ASCII, it should be a multibyte character, calculate the number of bytes {if (chr>=0x80 {if (CHR&GT;=0XFC&AMP;&AMP;CHR&LT;=0XFD) Nbytes=6;else if (Chr>=0xf8) Nbytes=5;else if (chr>=0xf0) nBytes=4; else if (CHR&GT;=0XE0) Nbytes=3;else if (chr>=0xc0) Nbytes=2;else{return FALSE;} nbytes--;}} else//The non-first byte of the multibyte character, which should be 10xxxxxx{if ((chr&0xc0)! = 0x80) {return FALSE;} nbytes--;}} if (nbytes > 0)//Counter-return rule {return FALSE;} if (BALLASCII)//If all are ASCII, the description is not Utf-8{return FALSE;} return TRUE;}

C + + determines whether a string is UTF8 encoded