What is encoding
The conversion of plaintext to encoded text is called "encoding", and it is "decoded" from the encoded text back to Cheng Mingwen.
Ascii
ASCII code:256 different states, each state is unique to one character, such as a--->00010001, and English only 26 characters, count some special characters and numbers, 128 states are enough; each level is called a bit, The 8 bits are agreed to form a byte so that the computer can store English text in 127 different bytes. This is
GB2312
ASCII for China has a problem, that is no way to identify Chinese, so the ASCII code is re-written again, that is, the formation of the ASCII Chinese extension, about the support of more than 7,000 Simplified Chinese characters
GBK and GB18030 Codes
Due to the limited number of Chinese supported by GB2312, the extended encoding scheme is called GBK Standard, and GBK includes all the contents of GB2312 while adding nearly 20,000 new Chinese characters (including traditional characters) and symbols.
UNICODE Coding
After a lot of coding, also raised a problem, is the language between the languages of the Code is not universal, computer garbled phenomenon, in order to solve this problem, the international standard who organization of the code is unified, and the unified code is called Unicode encoding.
Unicode is represented as a character in two bytes, and it can combine 65535 different characters in total, which is sufficient to cover all languages and symbols (including Oracle) in the world.
Example:
Variable:s="I‘m 苑昊"
Unicode mode (UCS-2) storage method:
I 00000000 01001001
' 00000000 00100111
M 00000000 01101101
00000000 00100000
Court 10000010 11010001
Hao 01100110 00001010
Utf8
UTF-8 (8-bit Unicode Transformation Format) is a variable-length character encoding for Unicode, which can represent a symbol using 1~4 bytes, depending on
Different symbols change the byte length, when the character is in the ASCII code range, it is represented by a byte, so it is compatible with ASCII-encoded
Unicode is a memory-encoded representation scheme (a specification), and UTF is a scheme for how to save and transmit Unicode (implementation), which is also the difference between UTF and Unicode.
Example:
Variable:s="I‘m 苑昊"
How to store UTF-8:
I 01001001
' 00100111
M 01101101
00100000
Court 11101000 10001011 10010001
Hao 11100110 10011000 10001010
The knowledge points that must be known:
1. In Python2 the default encoding is ASCII, the default is Unicode in Python3
2.unicode is divided into utf-32 (4 bytes), utf-16 (accounting for two bytes), Utf-8 (1-4 bytes), so utf-16 is now the most commonly used Unicode version, but in the file is still utf-8, because the UTF8 save space
3. Encode in Py3, while transcoding will also change the string to bytes type, decode decoding will also turn bytes back to string
converting a flowchart between encodings: (for Py2 only)
Encoding Conversion instance:
based on python2.x
Import Sys
Print (Sys.getdefaultencoding ()) #打印系统默认的编码
msg = "Hi, Baby"
msg_gb2312 = Msg.decode ("Utf-8"). Encode ("gb2312")
GB2312_TO_GBK = Msg_gb2312.decode ("GBK"). Encode ("GBK")
Print (msg)
Print (msg_gb2312)
Print (GB2312_TO_GBK)
Based on python3.x
Import Sys
Print (sys.getdefaultencoding ())
msg = "Hi, Baby"
#msg_gb2312 = Msg.decode ("Utf-8"). Encode ("gb2312")
msg_gb2312 = Msg.encode ("gb2312") #系统默认就是unicode, no more decode
Gb2312_to_unicode = Msg_gb2312.decode ("gb2312")
Gb2312_to_utf8 = Msg_gb2312.decode ("gb2312"). Encode ("Utf-8")
Print (msg)
Print (msg_gb2312)
Print (Gb2312_to_unicode)
Print (Gb2312_to_utf8)
python-character encoding