Unicode Character Set and encoding method, unicode Character Set Encoding

Unicode Character Set and encoding method, unicode Character Set Encoding

Generally, a set of all characters that can be expressed in a standard is called a character set. For example, the character set defined by ISO/Unicode is Unicode. In Unicode, each character occupies a single bitwise/Unicode number (expressed in 4-digit hexadecimal notation, Code point: U + FFFF), for example, U + 0000 is "Null ", U + 597D = "good ". The Unicode Character Set defines 114 to 112 such bits. it uniquely represents almost all characters in the world using the hexadecimal number from 0 to 10FFFF.

Although the bitwise in the character set is unique, because the computer stores data in bytesASCII (0x00-0x7F)To save storage space and other reasons, a specific encoding method is required to identify the bytecode bit. This rule specifies how the Unicode numbers of each character are stored (one or more bytes are used for storage and which bytes are used for storage). It is called "encoding ". Common Unicode Character Set-based encoding methods include UTF-8, UTF-16, and UTF-32.

The characters in the character set are mapped to the Unicode numbers one by one. The Unicode numbers are encoded into binary and stored in the memory. In the same binary segment, each byte has a serial number or a serial number for each two bytes. The decoding method is different, the resulting serial number is different, and the corresponding ing characters are also different. This is the cause of garbled characters.

Encoding method:

UTF-8 is a variable length encoding method, using the high reserve mode to distinguish different variable length. It uses 1 ~ The four bytes indicate a symbol, and the length of the byte varies according to different symbols. The more common the characters are, the shorter the byte, the first 128 characters are represented by only one byte, which is exactly the same as the ASCII code.

UTF-8 encoding rules:

1) for a single-byte symbol, the first byte is set to 0, and the last seven digits are the unicode code of this symbol. Therefore, for English letters, the UTF-8 encoding and ASCII code are the same.

2) for the N-byte symbol, the first N bits of the first byte are set to 1, the N + 1 bits are 0, and the first two bits of the next byte are all set to 10. The remaining binary digits are all unicode codes of this symbol.

Unicode symbol range | UTF-8 encoding method
(Hexadecimal) | (Binary)
----------- + ---------------
0000 0000-0000 007F | 0 xxxxxxx
0000 0080-0000 07FF | 110 xxxxx 10 xxxxxx
0000 0800-0000 FFFF | 1110 xxxx 10 xxxxxx 10 xxxxxx
0001 0000-0010 FFFF | 11110xxx 10 xxxxxx 10 xxxxxx 10 xxxxxx

If the first byte is 0, the byte is a single character. If the first byte is 1, the number of consecutive 1 represents the number of bytes occupied by the current character.

Because of the space-saving feature of UTF-8, it becomes the most common webpage code on the Internet.

Unicode Character Set https://unicode-table.com/en/#control-character.