Utf8 Chinese encoding range

The UTF-8 is somewhat similar to the Haffman encoding, which encodes Unicode into: bytes -0000007f characters, expressed in a single byte; bytes 0080-000007ff characters are expressed in two bytes (Chinese encoding range) the characters of 00000800-0000ffff are encoded and converted in 3 bytes: iconv-f "current file encoding"-T "file converted encoding"-o "new file name generated after conversion" "source file name" temp = iconv. conv ("UTF-8", "gb2312", a) since so far the Unicode-16 specification does not specify FFFF or more characters, the UTF-8 uses up to three bytes to represent a single character. Theoretically speaking, however, a UTF-8 needs a maximum of 6 bytes to represent a character. In the UTF-8, English characters are still the same as ASCII encoding, so the original function library can continue to use. The encoding range of Chinese characters is between 0080-07ff, so it is represented by two bytes (but the two bytes are different from those of GB encoding ). 0, big endian, little endianbig endian, and littleendian are different ways for the CPU to process the number of multi-word segments. For example, the Unicode code of the Chinese character is 6c49. When I write a file, do I write 6C in front or 49 in front? If you write 6C in front, it is bigendian. Write 49 in front, that is, little endian. The word "endian" comes from Gulliver Travel Notes. The civil war in the minor people's country originated from the fact that the big-Endian attack or the little-Endian attack were initiated when the eggs were eaten. As a result, there were six rebels, one of the emperors gave life and the other lost the throne. We generally translate endian into byte order, and call Big endian and littleendian "Big tail" and "Small Tail ". 4, UTF Encoding UTF-8 is to 8 bits as the unit to encode the UCS. The encoding from UCS-2 to UTF-8 is as follows: UCS-2 encoding (hexadecimal) UTF-8 byte stream (Binary) 0000-007f 0xxxxxxx0080-07ff 110 XXXXX 10xxxxxx0800-FFFF 1110 XXXX 10 xxxxxx 10xxxxxx for example, the Unicode encoding of the word "Han" is 6c49. 6c49 is between 0800-ffff, so it must use a 3-byte template: 1110xxxx10xxxxxx 10 xxxxxx. Write 6c49 as binary: 0110 110001 001001. Use this bit stream to replace X in the template. The result is 11100110 10110001 10001001, that is, E6 B1 89. Readers can use NotePad to test whether our encoding is correct. The UTF-16 is encoded in 16 bits. The UTF-16 code is equal to the 16-bit unsigned integer corresponding to the UCS code for a UCS code smaller than 0x10000. Defines Algorithm . However, because the actual use of ucs2, or ucs4 BMP must be less than 0x10000, so for now, it can be considered that the UTF-16 and UCS-2 is basically the same. But UCS-2 is only a coding scheme, UTF-16 is used for actual transmission, so we have to consider the problem of byte order. 5. UTF's byte order and BOMUTF-8 are encoded in bytes, without the problem of the byte order. The UTF-16 uses two bytes as the encoding unit. before interpreting a UTF-16 text, you must first understand the byte order of each encoding unit. For example, if the Unicode encoding of "queue" is 594e and that of "B" is 4e59. If we receive the UTF-16 byte stream "594e", is this "Kui" or "B "? The recommended method for marking byte order in Unicode specifications is Bom. Bom is not a "Bill ofmaterial" Bom, but a byte order mark. Bom is a bit clever: There is a character in the UCS code called "Zero Width NO-BREAKSPACE", which is encoded as feff. Fffe does not exist in the UCs, so it should not appear in actual transmission. We recommend that you transmit the character "zerowidth no-break space" before transmitting the byte stream in the UCS specification ". In this way, if the receiver receives feff, it indicates that the byte stream is big-Endian; if it receives fffe, it indicates that the byte stream is little-Endian. Therefore, the character "zerowidth no-break space" is also called Bom. The UTF-8 does not need BOM to indicate the byte order, but BOM can be used to indicate the encoding method. The UTF-8 code for the character "zerowidth no-break space" is EF bbbf (the reader can verify it using the encoding method we introduced earlier ). So if the receiver receives a byte stream starting with EF bbbf, it will know that this is UTF-8 encoding. Windows uses BOM to mark the encoding of text files.