Coding knowledge study Note 3

 

Coding knowledge study Note 3

I. How to code UTF-8

The UTF-8 is coded in 8 bits. The encoding from UCS-2 to UTF-8 is as follows:

Serial number	UCS-2 coding range (hexadecimal)	UTF-8 byte stream (Binary)	Description
1	0000-007f	0 xxxxxxx	1 byte in the format 0 xxxxxxx
2	0080-07ff	110 XXXXX 10 xxxxxx	Two bytes in the format 110 XXXXX 10 xxxxxx
3	0800-FFFF	1110 XXXX 10 xxxxxx 10 xxxxxx	Three bytes in the format: 1110 xxxx 10 xxxxxx 10 xxxxxx

2. Examples of UTF-8 Coding

For example, the Unicode code of the Chinese character is 6c49. 6c49 is between 0800-ffff, so it must use a 3-byte template: 1110 XXXX 10 xxxxxx 10 xxxxxx.

Write 6c49 as binary: 0110 110001 001001. Use this bit stream to replace X in the template. The result is 11100110 10110001 10001001, that is, E6 B1 89.

3. How to code UTF-16

UTF-16 is encoded in 16 bits.

The UTF-16 code is equal to the 16-bit unsigned integer corresponding to the UCS code for a UCS code that is less than 0x10000.

An algorithm is defined for an UCS code greater than or equal to 0x10000.

However, because the actual use of ucs2 or the BMP of ucs4 must be smaller than 0x10000,UTF-16 can be considered to be basically the same as the UCS-2.

4. Differences between UTF-16 and UCS-2

UCS-2 is just a coding scheme, but UTF-16 is used for actual transmission, so we have to consider the issue of byte order.

5. Why is there a byte order in UTF Encoding?

UTF-8 uses bytes as the encoding unit, so there is no bytecode problem.

The UTF-16 uses two bytes as the encoding unit. before interpreting a UTF-16 text, you must first understand the byte order of each encoding unit. For example, if the Unicode encoding of "queue" is 594e and that of "B" is 4e59. If we receive the UTF-16 byte stream "594e", is this "Kui" or "B "?

6. UFT collation

The UTF-8 has no collation issues

The problem of UTF-16 in byte order

7. How to Solve the Problem of UTF-16 byte order

The recommended method for marking byte order in Unicode specifications is Bom. Bom is not a "bill of material" Bom, but a byte order mark. Bom method:

 

There is a character named "Zero Width no-break space" in the UCS encoding, and its encoding is feff. Fffe does not exist in the UCs, so it should not appear in actual transmission. We recommend that you transmit the character "Zero Width no-break space" before transmitting the byte stream in the UCS specification ".

 

In this way, if the receiver receives feff, it indicates that the byte stream is big-Endian; if it receives fffe, it indicates that the byte stream is little-Endian. Therefore, the character "Zero Width no-break space" is also called Bom.

 

7. methods to solve the problem of UTF-16 byte order:

BOM

Bom is the abbreviation of byte order mark

8. Bom Implementation ideas:

There is a character named "Zero Width no-break space" in the UCS encoding, and its encoding is feff. Fffe does not exist in the UCs, so it should not appear in actual transmission. We recommend that you transmit the character "Zero Width no-break space" before transmitting the byte stream in the UCS specification ".

 

In this way, if the receiver receives feff, it indicates that the byte stream is big-Endian; if it receives fffe, it indicates that the byte stream is little-Endian. Therefore, the character "Zero Width no-break space" is also called Bom.

9. What is the significance of BOM to UTF-8?

UTF-8 does not require BOM to indicate the byte order, but BOM can be used to indicate the encoding method.

The UTF-8 code for the character "Zero Width no-break space" is EF BB BF

So if the receiver receives a byte stream starting with ef bb bf, it will know that this is UTF-8 encoding.