C + + instance: Determine if the string is UTF8 encoded

Source: Internet
Author: User

ref:http://hi.baidu.com/lovevc2008/item/2b69a7caf224e62946d5c093

UTF8 is the most basic unit of 8bits, 1Bytes, and of course it can be based on 16bits and 32bits, which are called UTF16 and UTF32, but are not used much, and UTF8 is widely used in file storage and network transmission.
Coding principle
Look at this template first:
UCS-4 range (hex.) UTF-8 octet sequence (binary)
0000 0000-0000 007F 0xxxxxxx
0000 0080-0000 07FF 110xxxxx 10xxxxxx
0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-001f FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03ff FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7fff FFFF 1111110x 10xxxxxx ... 10xxxxxx
Encoding Step:
1 First determine how many 8bits (octets) you need
2 to populate each octets's high bits with the above template
3 to fill the bits of the characters into X, character order: Low → High, UTF8 order: The last octet the lowest x→ the first octet the highest bit X
Based on UTF8 encoding, up to 6 bytes, so UTF8 is a 1-6-byte code
C + + code is as follows:
int IsTextUTF8 (char* str,ulonglong length)
{
int i;
DWORD NBYTES=0;//UFT8 can be encoded in 1-6 bytes, ASCII with one byte
Uchar Chr;
BOOL ballascii=true; If all is ASCII, the description is not UTF-8
for (i=0;i<length;i++)
{
chr= * (str+i);
if ((chr&0x80)!= 0)//To determine whether ASCII encoding, if not, is likely to be utf-8,ascii with 7-bit encoding, but with a byte deposit, the highest bit marked as 0,o0xxxxxxx
Ballascii= FALSE;
if (nbytes==0)//If not ASCII, it should be a multi-byte character, count the number of bytes
{
if (chr>=0x80)
{
if (CHR&GT;=0XFC&AMP;&AMP;CHR&LT;=0XFD)
nbytes=6;
else if (CHR&GT;=0XF8)
nbytes=5;
else if (chr>=0xf0)
nbytes=4;
else if (CHR&GT;=0XE0)
nbytes=3;
else if (CHR&GT;=0XC0)
nbytes=2;
Else
{
return FALSE;
}
nbytes--;
}
}
else//multibyte character is not the first byte, should be 10xxxxxx
{
if ((chr&0xc0)!= 0x80)
{
return FALSE;
}
nbytes--;
}
}
if (nbytes > 0)//violation rules
{
return FALSE;
}
if (BALLASCII)//If all are ASCII, the description is not UTF-8
{
return FALSE;
}
return TRUE;
}

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.