[Learning] Chinese character encoding and related issues

Transfer

In a computer, the character is generally not saved as an image. Each character is represented by an encoding, And the encoding used by each character depends on the character set used ).
At the beginning, there was only one character set on the Internet-the ansi ascii character set, which uses 7 bits to represent a single character, representing a total of 128 characters, it contains frequently used characters such as English letters, numbers, and punctuation marks. Then, it is extended. 8 bits represents a single character and can represent 256 characters. Some special characters such as tabs are added to the original 7 bits character set.
Later, because of the addition of languages in various countries, ASCII could no longer meet the needs of information exchange. Therefore, in order to express the texts of other countries, countries developed their own character sets based on ASCII, these character sets derived from the ANSI standard are commonly referred to as the ANSI character set. Their formal names should be MBCS (Multi-Byte Chactacter System, that is, Multi-Byte Character System ). These derived character sets are based on ASCII 127 bits and compatible with ASCII 127. They use encoding greater than 128 as a Leading Byte, followed by the second (or even third) after Leading Byte) character and Leading Byte are used as the actual encoding. There are many such character sets, and our common GB-2312 is one of them.
For example, in the GB-2312 character set, "connectivity" is encoded as C1 ac cd A8, where C1 and CD are Leading bytes. The first 127 encodings are retained as standard ASCII. For example, the "0" encoding is 30 H (30 H indicates the hexadecimal 30 ). When the software reads data, if it sees 30 H, it knows that it is less than 128 is the standard ASCII, indicating "0". If it sees C1 greater than 128, it will know that there is another code behind it, so C1 AC forms an entire encoding, which represents a "connection" in the GB-2312 character set ".
Because each language has its own character set, there are too many character sets in the end, and it is inconvenient to convert character sets frequently during international exchanges. Therefore, the Unicode Character Set is proposed. It uses 16 bits (two bytes, one word) to represent a character, which can represent 65536 characters in total. Including common characters in almost all languages in the world, facilitating information exchange. Standard Unicode is called a UTF-16. Later, in order to enable the dual-byte Unicode to be correctly transmitted in the existing single-byte processing system, a UTF-8 emerged and Unicode was encoded in a way similar to MBCS. Note that the UTF-8 is an encoding that belongs to the Unicode Character Set. Unicode character sets can be encoded in multiple forms, while ASCII has only one type, and most MBCS (including GB-2312) have only one.
For example, the Unicode Standard UTF-16 (big endian) of the Connected Word is: DE 8F 1A 90
And its UTF-8 code: E8 BF 9E E9 80 9A
Finally, when a software opens a text, the first thing it needs to do is to decide which encoding of the text is used and saved. The software has three ways to determine the character set and encoding of the text:
The most standard way is to check the first several bytes of the text, as shown in the following table:

Character Set/encoding
Ef bb bf UTF-8
Fe ff UTF-16/UCS-2, little endian
Ff fe UTF-16/UCS-2, big endian
Ff fe 00 00 UTF-32/UCS-4, little endian.
00 00 fe ff UTF-32/UCS-4, big-endian.

For example, after inserting a tag, the UTF-16 (big endian) that connects the word "and the UTF-8 Code are:
Ff fe de 8F 1A 90
Ef bb bf E8 BF 9E E9 80 9A
However, the MBCS text does not contain these Character Set tags that start with. Unfortunately, some early and poorly designed software does not insert these Character Set tags that start with when saving Unicode text. Therefore, software cannot rely on this approach. In this case, the software can determine the character set and Its Encoding in a safer way, that is, a dialog box is displayed to ask the user, for example, dragging the "connected" file to MS Word, the Word dialog box is displayed.
If the software does not want to bother users, or it is not convenient to ask users, it can only "Guess" its own method. The software can guess which charset it may belong to based on the characteristics of the entire text, this is probably not allowed. Use NotePad to open the connected file.
We can prove this: After you type "Connect" in notepad and select "Save As", you will see that "ANSI" is displayed in the last drop-down box, and Save it. When you open the connected File garbled, and then click "File"-> "Save As", you will see the last drop-down box shows "UTF-8 ", this indicates that notepad considers the text currently opened as a UTF-8-encoded text. We saved the data in the ANSI character set just now. This shows that notepad guesses the character set of the "connected" file and thinks it is more like a UTF-8-encoded text. This is because the GB-2312 code of the connected word looks more like the UTF-8 code, which is a coincidence that not all words are like this. You can use the notepad OPEN function to open the "Connect" file and select ANSI in the last drop-down box to display it normally. Conversely, if it was previously saved as a UTF-8 encoding, it would not be a problem to open it directly.
If you put the connected file in MS Word, Word will also think it is a UTF-8-encoded file, but it cannot be determined, so a dialog box will pop up asking the user, select "simplified Chinese (GB2312)" to enable it. Notepad is simplified at this point, which is consistent with the positioning of this program.
Once again, we would like to thank the senior engineers for their explanations and a clear understanding of this phenomenon.