Coding Problems of Python3

Python3 a clearer distinction between text (str) and binary data (bytes).

Text is encoded by default in Unicode (Python2 is ASCII by default), denoted by the STR type, and binary data is represented by the bytes type.

str=' Chinese 中文版'

STR is a text type, that is, the STR type

 >>> Str.encode ( " utf-8  Span style= "color: #800000;" > '   ' b   "  \xe4\xb8\xad\xe6\x96\x87english   " >>> Str.encode ( '   >>> bytes (str, " utf-8   " ) b   \xe4\xb8\xad\xe6\x96\x87english   ' 

The bytes () function is the same as Str.encode (), which encodes the str type as the bytes type

>>> b'\xe4\xb8\xad\xe6\x96\x87english'. Decode ('utf-8')  )' Chinese 中文版'

The decoding process, which translates the bytes data into str

>>> b'\xe4\xb8\xad\xe6\x96\x87english'. Encode ('Utf-8') Traceback (most recent): File"<pyshell#42>", Line 1,inch<module>b'\xe4\xb8\xad\xe6\x96\x87english'. Encode ('Utf-8') Attributeerror:'bytes'object has no attribute'encode'

The bytes data cannot continue to be encoded as bytes

>>>'Chinese 中文版'. Decode ('Utf-8') Traceback (most recent): File"<pyshell#44>", Line 1,inch<module>'Chinese 中文版'. Decode ('Utf-8') Attributeerror:'Str'object has no attribute'Decode'

You cannot continue to decode STR data to Str.

That is, the encoding process is from Str to bytes, and the decoding process is from bytes to Str.

>>> b'\xe4\xb8\xad\xe6\x96\x87english'. Decode ('gb2312') Traceback (most recent): File"<pyshell#45>", Line 1,inch<module>b'\xe4\xb8\xad\xe6\x96\x87english'. Decode ('gb2312') Unicodedecodeerror:'gb2312'Codec can'T decode byte 0xad in position 2:illegal multibyte sequence

Above is the Utf-8 encoded bytes in gb2312 way decoding, the result is wrong, because there is no corresponding gb2312 encoding 0xad

What if you want to know what Unicode encoding a string of bytes codes is? This is actually not absolutely certain, otherwise garbled will not happen.

Third-party library Chardet, using the function detect can "guess" the encoding method.

From Chardet Import detect
>>> Detect (b'\xe4\xb8\xad\xe6\x96\x87english') {'  Confidence'encoding'utf-8'}

Here the confidence 0.7525, can be simply understood as probability 0.7525, here there are only two Chinese characters, if bytes long enough, then the confidence must be higher

>>> Detect (b'\xe6\x88\x91\xe6\x98\xaf\xe4\xb8\xad\xe6\x96\x87\xe6\x88\x91\xe7\x9c\x9f\xe7 \x9a\x84\xe6\x98\xaf\xe4\xb8\xad\xe6\x96\x87') {'confidence' ' encoding ' ' Utf-8 '}

Here are 10 Chinese characters, and the result is 0.99.

Coding Problems of Python3