python-character encoding

Source: Internet
Author: User

What is encoding

The conversion of plaintext to encoded text is called "encoding", and it is "decoded" from the encoded text back to Cheng Mingwen.

Ascii

ASCII code:256 different states, each state is unique to one character, such as a--->00010001, and English only 26 characters, count some special characters and numbers, 128 states are enough; each level is called a bit, The 8 bits are agreed to form a byte so that the computer can store English text in 127 different bytes. This is

GB2312

ASCII for China has a problem, that is no way to identify Chinese, so the ASCII code is re-written again, that is, the formation of the ASCII Chinese extension, about the support of more than 7,000 Simplified Chinese characters

GBK and GB18030 Codes

Due to the limited number of Chinese supported by GB2312, the extended encoding scheme is called GBK Standard, and GBK includes all the contents of GB2312 while adding nearly 20,000 new Chinese characters (including traditional characters) and symbols.

UNICODE Coding
After a lot of coding, also raised a problem, is the language between the languages of the Code is not universal, computer garbled phenomenon, in order to solve this problem, the international standard who organization of the code is unified, and the unified code is called Unicode encoding.
Unicode is represented as a character in two bytes, and it can combine 65535 different characters in total, which is sufficient to cover all languages and symbols (including Oracle) in the world.

Example:

Variable:s="I‘m 苑昊"

Unicode mode (UCS-2) storage method:

I 00000000 01001001
' 00000000 00100111
M 00000000 01101101
00000000 00100000
Court 10000010 11010001
Hao 01100110 00001010

Utf8
UTF-8 (8-bit Unicode Transformation Format) is a variable-length character encoding for Unicode, which can represent a symbol using 1~4 bytes, depending on
Different symbols change the byte length, when the character is in the ASCII code range, it is represented by a byte, so it is compatible with ASCII-encoded
Unicode is a memory-encoded representation scheme (a specification), and UTF is a scheme for how to save and transmit Unicode (implementation), which is also the difference between UTF and Unicode.

Example:

Variable:s="I‘m 苑昊"

How to store UTF-8:

I 01001001
' 00100111
M 01101101
00100000
Court 11101000 10001011 10010001
Hao 11100110 10011000 10001010

The knowledge points that must be known:

1. In Python2 the default encoding is ASCII, the default is Unicode in Python3

2.unicode is divided into utf-32 (4 bytes), utf-16 (accounting for two bytes), Utf-8 (1-4 bytes), so utf-16 is now the most commonly used Unicode version, but in the file is still utf-8, because the UTF8 save space

3. Encode in Py3, while transcoding will also change the string to bytes type, decode decoding will also turn bytes back to string

converting a flowchart between encodings: (for Py2 only)

Encoding Conversion instance:

based on python2.x

Import Sys
Print (Sys.getdefaultencoding ()) #打印系统默认的编码
msg = "Hi, Baby"
msg_gb2312 = Msg.decode ("Utf-8"). Encode ("gb2312")
GB2312_TO_GBK = Msg_gb2312.decode ("GBK"). Encode ("GBK")
Print (msg)
Print (msg_gb2312)
Print (GB2312_TO_GBK)

Based on python3.x

Import Sys

Print (sys.getdefaultencoding ())

msg = "Hi, Baby"

#msg_gb2312 = Msg.decode ("Utf-8"). Encode ("gb2312")

msg_gb2312 = Msg.encode ("gb2312") #系统默认就是unicode, no more decode

Gb2312_to_unicode = Msg_gb2312.decode ("gb2312")

Gb2312_to_utf8 = Msg_gb2312.decode ("gb2312"). Encode ("Utf-8")

Print (msg)

Print (msg_gb2312)

Print (Gb2312_to_unicode)

Print (Gb2312_to_utf8)

python-character encoding

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.