Python notes-character encoding and Transcoding

Source: Internet
Author: User

Python notes-character encoding and Transcoding

I. Prerequisites

So what is encoding? // Remember one sentence in ASCII: All data in the computer, whether it is text, images, videos, or audio files, is essentially stored in binary format similar to 01010101. Simply put, computers only know binary numbers! So the purpose is clear: how can we uniquely match a group of binary numbers that we can recognize? As a result, the comrades in the United States think that 0 or 1 is represented by a level of high and low States. If the eight levels are regarded as a group, 256 different states are displayed. Each State corresponds to only one character, for example, A ---> 00010001, while English only has 26 characters, some special characters and numbers are counted, and the 128 States are enough. Each level is called A bit, 8 bits constitute a byte, so that the computer can use 127 different bytes to store English text. This is the ASCII code. Extended ANSI encoding: At first, a byte has eight bits, but the maximum bit is useless. The default value is 0. Later, it can also represent Latin for the computer, the last character set is also used. The character set from 128 to 255 corresponds to the Latin. So far, one byte is full! // When the GB2312 computer came across the ocean and came to China, the problem came. The computer did not know Chinese, and of course it could not display Chinese. In addition, all the statuses of one byte were full, the heart of my evil imperialism is not dead! Our party is also great and self-reliant. We have to rewrite a table and delete all the latans corresponding to the extended eighth digit. It is stipulated that the meaning of a character smaller than 127 is the same as that of the original one, but when two characters above 127 are connected together, it indicates a Chinese character. The first byte (also known as the high byte) uses 0xF7 from 0xA1, followed by a byte (low byte) from 0xA1 to 0xFE, we can combine more than 7000 simplified Chinese characters. This Chinese Character scheme is called "GB2312 ". GB2312 is an extension of Chinese ASCII. // GBK and GB18030 are encoded but there are too many Chinese characters and GB2312 is not enough. Therefore, it is specified that if the first byte is greater than 127, it indicates that this is the beginning of a Chinese character, whether the content in the extended character set is followed or not. The expanded encoding scheme is called the GBK standard. GBK includes all the content of GB2312, and adds nearly 20000 New Chinese characters (including traditional Chinese characters) and symbols. // UNICODE encoding: many other countries develop their own encoding standards, but they do not support each other. This poses many problems. Therefore, the Standardization Organization for unified encoding: The standard encoding criterion is UNICODE. UNICODE is expressed as a character in two bytes. It can combine a total of 65535 different characters, which is sufficient to cover all the symbols in the world (including Oracle) // utf8: unicode is all over the world. Why do we need utf8 encoding? Everyone thinks that for people in the English world, one byte is enough. For example, if we want to store A, we can use 00010001 bytes. Now we have A unicode dinner, and we have to use two bytes: 00000000 million RMB, too much waste! Based on this, scientists in the United States proposed the idea of genius: utf8. UTF-8 (8-bit Unicode Transformation Format) is a variable-length character encoding for Unicode, it can use 1 ~ The four bytes indicate one character. The length varies according to different characters. When the character is within the ASCII code range, it is represented by one byte. Therefore, it is compatible with ASCII encoding. The obvious advantage is that although the data in our memory is unicode, when the data is to be saved to the disk or used for network transmission, unicode is far less space than utf8! That's why utf8 is our recommended encoding method. The relationship between Unicode and utf8: Unicode is a memory encoding Representation Scheme (standard), and how UTF saves and transfers Unicode (Implementation) this is also the difference between UTF and Unicode.

 

We only need to know that unicode is a universal code, while UTF-8 and gbk are extended From unicode, So we convert UTF-8 to gbk in the form of converting to unicode and then to gbk; similarly, if other encodings are converted to UTF-8, they are converted to unicode first and then UTF-8.

 

The python3 string is Unicode encoded, and the text is always Unicode, represented by the str type, and the binary data is represented by the bytes type.

Which are easy to misunderstand:

(1) In py3, the default file encoding is UTF-8, so you can directly write Chinese characters without the file header declaration encoding.
(2) The variables you declare are unicode encoded by default. Even if your file header declares that the encoding is UTF-8, they are not UTF-8, because the default is unicode.

Import sys
Print (sys. getdefaultencoding () # print the default encoding of the system

# S = u "" # Add a utable to show that upython uses the unicodenicode encoding format by default. The output result is the same as that without u, indicating that python uses unicode by default.
S = "hello" #, that is, this s is the unicode encoding format
Print (s. encode ("UTF-8") # encode decoding tells the system s to convert to UTF-8 and Output A byte object.
Print (s. encode ("UTF-8"). decode ("gbk") # indicates transcoding to the gbk format
Print (s. encode ("gbk "))
Print (s. encode ("UTF-8"). decode ("UTF-8"). encode ("gbk "))
Print (s. encode () # UTF-8 is not written by default.
Print (s. encode ("UTF-8"). decode ())
Output result:

>>> UTF-8
>>> B '\ xe4 \ xbd \ xa0 \ xe5 \ xa5 \ xbd' # B 'indicates the byte type
>>> Zookeeper >>> B '\ xc4 \ xe3 \ xba \ xc3'
>>> B '\ xc4 \ xe3 \ xba \ xc3'
>>> B '\ xe4 \ xbd \ xa0 \ xe5 \ xa5 \ xbd'
>>> Hello

 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.