Preface
Recently, due to project needs, you need to read a TXT document containing Chinese, and then save the file. The document was previously encoded by Base64, resulting in garbled reading of all Chinese characters. After the project team abandoned Base64, there were two errors:
ASCII codec can ' t encode characters in position ordinal not in range 128UnicodeDecodeError: ' UTF8 ' codec can ' t decode byte 0x.
If you are not familiar with ASCII, Unicode, and utf-8, you can look at the previous article about strings and encodings
Then there must be some understanding of the following three concepts:
-
ASCII can only represent numbers, English letters and some special symbols, and cannot represent Chinese characters
-
Both Unicode and utf-8 can represent Chinese characters, Unicode is fixed length, utf-8 is variable length
-
In-memory storage is generally Unicode, and disk file storage is generally utf-8, because utf-8 can save storage space
So what is the default code for Python?
>>> import sys>>> sys.getdefaultencoding () ' ASCII ' >>> reload (SYS) <module ' sys ' ( Built-in) >>>> sys.setdefaultencoding (' utf-8 ') >>> sys.getdefaultencoding () ' Utf-8 '
The default encoding for Python is ASCII, which allows you to sys.setdefaultencoding('utf-8') set Python's default encoding by using a function.
Python can change the encoding of the data by encode and decode, such as:
>>> u ' kanji ' u ' \u6c49\u5b57 ' >>> u ' kanji '. Encode (' utf-8 ') ' \xe6\xb1\x89\xe5\xad\x97 ' >>> u ' kanji '. Encode (' Utf-8 '). Decode (' Utf-8 ') u ' \u6c49\u5b57 '
We can set the encoding by these two functions.
So, what type of STR is in Python?
>>> import binascii>>> ' kanji ' \xba\xba\xd7\xd6 ' >>> type (' kanji ') <type ' str ' >>> > Print binascii.b2a_hex (' kanji ') babad7d6>>> print binascii.b2a_hex (U ' kanji ') Traceback (most recent call last): File "<stdin>", line 1, in <module>unicodeencodeerror: ' ASCII ' codec can ' t encode characters inposition 0-1: o Rdinal not in range >>> print binascii.b2a_hex (U ' kanji '. Encode (' Utf-8 ')) e6b189e5ad97>>> print Binascii.b2a_hex (U ' kanji '. Encode (' GBK ')) babad7d6
Binascii is the binary conversion of data into ASCII, the above explanation is: ' Kanji ' type is str, binary is babad7d6,u ' kanji ' is unable to convert to ASCII, so that the first error is reported. The solution is to put it. Encode (' utf-8 ') into STR type. Because my command line is Windows default GBK encoding, all U ' Kanji ' .encode(‘gbk') , the output result is the same as the ' kanji ' result.
To summarize, Python's str is actually one of the Unicode, Python's default encoding is ASCII, for non-ASCII into ASCII will be error, keep in mind the following rules:
-
Unicode = encode (' appropriate code ') = = str
-
str = decode (' fit code ') = Unicode
There is also a simple way to set the code in the file header, you can save a lot of trouble:
Import sysreloads (SYS) sys.setdefaultencoding (' Utf-8 ')
For the second problem, it is the error when the file is read. Utf-8 file has a BOM and no BOM two ways, the difference seems to be in the BOM file than no BOM file more than a head, resulting in the utf-8 way to read the file times wrong, I have previously tried to read the document before the BOM to judge, skip the BOM file head, and then failed, really awkward ~ ~.
Also need Google to help God, the specific way is to use the codecs library to read the file (I guess this library is to detect the head of the file).
Import Codecscodecs.open (file_name, "R", encoding= ' utf-8 ', errors= ' ignore ')
For coding problems, be sure to understand how ASCII, Unicode, and utf-8 work.