ref:http://hi.baidu.com/lovevc2008/item/2b69a7caf224e62946d5c093
UTF8 is the most basic unit of 8bits, 1Bytes, and of course it can be based on 16bits and 32bits, which are called UTF16 and UTF32, but are not used much, and UTF8 is widely used in file storage and network transmission.
Coding principle
Look at this template first:
UCS-4 range (hex.) UTF-8 octet sequence (binary)
0000 0000-0000 007F 0xxxxxxx
0000 0080-0000 07FF 110xxxxx 10xxxxxx
0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-001f FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03ff FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7fff FFFF 1111110x 10xxxxxx ... 10xxxxxx
Encoding Step:
1 First determine how many 8bits (octets) you need
2 to populate each octets's high bits with the above template
3 to fill the bits of the characters into X, character order: Low → High, UTF8 order: The last octet the lowest x→ the first octet the highest bit X
Based on UTF8 encoding, up to 6 bytes, so UTF8 is a 1-6-byte code
C + + code is as follows:
int IsTextUTF8 (char* str,ulonglong length)
{
int i;
DWORD NBYTES=0;//UFT8 can be encoded in 1-6 bytes, ASCII with one byte
Uchar Chr;
BOOL ballascii=true; If all is ASCII, the description is not UTF-8
for (i=0;i<length;i++)
{
chr= * (str+i);
if ((chr&0x80)!= 0)//To determine whether ASCII encoding, if not, is likely to be utf-8,ascii with 7-bit encoding, but with a byte deposit, the highest bit marked as 0,o0xxxxxxx
Ballascii= FALSE;
if (nbytes==0)//If not ASCII, it should be a multi-byte character, count the number of bytes
{
if (chr>=0x80)
{
if (CHR>=0XFC&&CHR<=0XFD)
nbytes=6;
else if (CHR>=0XF8)
nbytes=5;
else if (chr>=0xf0)
nbytes=4;
else if (CHR>=0XE0)
nbytes=3;
else if (CHR>=0XC0)
nbytes=2;
Else
{
return FALSE;
}
nbytes--;
}
}
else//multibyte character is not the first byte, should be 10xxxxxx
{
if ((chr&0xc0)!= 0x80)
{
return FALSE;
}
nbytes--;
}
}
if (nbytes > 0)//violation rules
{
return FALSE;
}
if (BALLASCII)//If all are ASCII, the description is not UTF-8
{
return FALSE;
}
return TRUE;
}