Python3 character encoding

Coding

A string is a data type, but a particular string is a coding problem.

Because a computer can only handle numbers, if you are working with text, you must convert the text to a number before processing it. The oldest computer was designed with 8 bits (bit) as a byte (byte), so a single word energy-saving representation of the largest integer is 255 (binary 11111111 = decimal 255), if you want to represent a larger integer, you must use more bytes. For example, two bytes can represent the largest integer is 65535 , 4 bytes can represent the largest integer is 4294967295 .

Since the computer was invented by the Americans, only 127 letters were encoded into the computer, that is, letters, numbers, and symbols, which are called ASCII encodings, such as uppercase letters encoded in A 65 lowercase letters z 122 .

Unicode

Unicode unifies all languages into a set of encodings, so there is no more garbled problem.

The Unicode standard is also evolving, but it is most commonly used to represent a character in two bytes (4 bytes If a very remote character is used). Unicode is supported directly by modern operating systems and most programming languages.

Now, the difference between ASCII encoding and Unicode encoding is smoothed:ASCII encoding is 1 bytes , and Unicode encoding is usually 2 bytes .

Letters A with ASCII encoding are decimal 65 , binary 01000001 ;

Characters 0 with ASCII encoding are decimal 48 , binary 00110000 , and note that ‘0‘ the characters and integers 0 are different;

Chinese characters are beyond the ASCII encoding range, Unicode encoding is decimal 20013 , binary 01001110 00101101 .

If you encode ASCII encoding in A Unicode, you only need to make 0 in front, so A the Unicode encoding is 00000000 01000001 .

The new problem arises again: If Unicode encoding is unified, the garbled problem disappears. However, if you write text that is basically all in English, using Unicode encoding requires more storage space than ASCII encoding, which is not cost-effective in storage and transmission.

Therefore, the encoding of converting Unicode encoding to "variable length encoding" has appeared UTF-8 . The UTF-8 encoding encodes a Unicode character into 1-6 bytes according to a different number size, the commonly used English letter is encoded in 1 bytes, the kanji is usually 3 bytes, and only the very uncommon characters are encoded into 4-6 bytes. If the text you want to transfer contains a large number of English characters, you can save space with UTF-8 encoding:

character	ASCII	Unicode	UTF-8
A	01000001	00000000 01000001	01000001
In	X	01001110 00101101	11100100 10111000 10101101

It can also be found from the table above that the UTF-8 encoding has an added benefit that ASCII encoding can actually be seen as part of the UTF-8 encoding, so a large number of legacy software that only supports ASCII encoding can continue to work under UTF-8 encoding.

Figuring out the relationship between ASCII, Unicode, and UTF-8, we can summarize how the current computer system works with character encoding:

In computer memory, Unicode encoding is used uniformly , and is converted to UTF-8 encoding when it needs to be saved to the hard disk or when it needs to be transferred.

When editing with Notepad, the UTF-8 characters read from the file are converted to Unicode characters into memory, and when the edits are complete, the conversion of Unicode to UTF-8 is saved to the file:

When you browse the Web, the server converts dynamically generated Unicode content to UTF-8 and then to the browser:

So you see a lot of pages of the source code will have similar <meta charset="UTF-8" /> information, that the page is exactly the UTF-8 encoding.

A Python string

In the latest version of Python 3, strings are encoded in Unicode, meaning that Python strings support multiple languages, such as:

Print (' str containing Chinese') contains the Chinese str

For the encoding of a single character, Python provides an ord() integer representation of the function to get the character, and the chr() function converts the encoding to the corresponding character:

>>> Ord ('A')65>>> ord (' Middle ') 20013>>> chr'B'>>> chr (25991)  ' wen '

If you know the integer encoding of a character, you can write it in hexadecimal.str

' \u4e2d\u6587 '//Chinese

Byte

Because the Python string type is str , in memory, in Unicode, one character corresponds to a number of bytes. If you want to transfer on a network, or save to disk, you need to turn it str into bytes bytes .

Python bytes uses b a prefixed single or double quotation mark for data of type:

x = B ' ABC '

要注意区分‘ABC‘和b‘ABC‘，前者是str，后者虽然内容显示得和前者一样，但bytes的每个字符都只占用一个字节。

The str pass method, expressed in Unicode encode() , can be encoded as specified bytes , for example:

>>>'ABC'. Encode ('ASCII') b'ABC'>>>'English'. Encode ('Utf-8') b'\xe4\xb8\xad\xe6\x96\x87'>>>'English'. Encode ('ASCII') Traceback (most recent): File"<stdin>", Line 1,inch<module>Unicodeencodeerror:'ASCII'Codec can'T encode characters in position 0-1: Ordinal not in range (+)

Pure English str can be ASCII encoded as bytes , content is the same, containing Chinese str can be UTF-8 encoded as bytes . strcannot be encoded in Chinese ASCII because the range of Chinese encoding exceeds the range of the ASCII encoding, and Python will make an error.

In bytes , the bytes that cannot be displayed as ASCII characters are \x## displayed.

Conversely, if we read the byte stream from the network or disk, then the data read is bytes . To turn bytes str it into, you need to use the decode() method:

>>> b'ABC'. Decode ('ascii')' ABC '>>> b'\xe4\xb8\xad\xe6\x96\x87'. Decode ('  Utf-8')' Chinese '

To calculate str how many characters are included, you can use a len() function

>>> Len ('ABC')3>>> len (' Chinese ' ) )2

len()The function calculates the str number of characters, and if bytes so, the len() function calculates the number of bytes

>>> Len (b'ABC')3>>> len (b'\xe4\xb8\xad\xe6\ x96\x87')6>>> len ( ' Chinese '. Encode ('  Utf-8'))6

1 Chinese characters are UTF-8 encoded and typically consume 3 bytes, while 1 English characters take up only 1 bytes.

We often encounter str and convert to and bytes from each other when manipulating strings. In order to avoid garbled problems, we should always adhere to the use of UTF-8 encoding str and bytes conversion.

The Python source code is also a text file, so when you save the source code, you need to be sure to save it as UTF-8 encoding when you include Chinese in your source. When the Python interpreter reads the source code, in order for it to be read by UTF-8 encoding, we usually write these two lines at the beginning of the file

# !/usr/bin/env Python3 # -*-coding:utf-8-*-

The second line of comments is to tell the Python interpreter to read the source code according to the UTF-8 encoding, otherwise the Chinese output you write in the source code may be garbled.

Formatting:

In Python, the format used is consistent with the C language, and is implemented as an % example:

Format% (.... params)

 >>>  hello,%s   ' %  '  world    '  hello, World   '  >>>  '  hi,%s, you have $%d.  % ("   Michael  , 1000000)   '  

%The operator is used to format the string. Inside the string, the representation is replaced by a %s string, %d represented by an integer substitution, %x representing a 16-decimal integer, with several %? placeholders, followed by several variables or values, in order to correspond well. If there is only one %? , the parentheses can be omitted.

formatting integers and floating-point numbers can also specify whether to complement 0 and the number of digits of integers and decimals:

' %2d-%02d ' % (3, 1)'  3-01'%.2f' % 3.1415926'3.14'

Sometimes, % what about a normal character inside a string? This time you need to escape and use it %% to represent one % :

' growth rate:%d percent ' % 7'growth rate:7%'

Python3 character encoding