Python solves Chinese character coding problem: Unicode Decode Error_python

Source: Internet
Author: User
Preface

Recently, due to project needs, you need to read a TXT document containing Chinese, and then save the file. The document was previously encoded by Base64, resulting in garbled reading of all Chinese characters. After the project team abandoned Base64, there were two errors:

ASCII codec can ' t encode characters in position ordinal not in range 128UnicodeDecodeError: ' UTF8 ' codec can ' t decode byte 0x.

If you are not familiar with ASCII, Unicode, and utf-8, you can look at the previous article about strings and encodings

Then there must be some understanding of the following three concepts:

    1. ASCII can only represent numbers, English letters and some special symbols, and cannot represent Chinese characters

    2. Both Unicode and utf-8 can represent Chinese characters, Unicode is fixed length, utf-8 is variable length

    3. In-memory storage is generally Unicode, and disk file storage is generally utf-8, because utf-8 can save storage space

So what is the default code for Python?

>>> import sys>>> sys.getdefaultencoding () ' ASCII ' >>> reload (SYS) <module ' sys ' ( Built-in) >>>> sys.setdefaultencoding (' utf-8 ') >>> sys.getdefaultencoding () ' Utf-8 '

The default encoding for Python is ASCII, which allows you to sys.setdefaultencoding('utf-8') set Python's default encoding by using a function.

Python can change the encoding of the data by encode and decode, such as:

>>> u ' kanji ' u ' \u6c49\u5b57 ' >>> u ' kanji '. Encode (' utf-8 ') ' \xe6\xb1\x89\xe5\xad\x97 ' >>> u ' kanji '. Encode (' Utf-8 '). Decode (' Utf-8 ') u ' \u6c49\u5b57 '

We can set the encoding by these two functions.

So, what type of STR is in Python?

>>> import binascii>>> ' kanji ' \xba\xba\xd7\xd6 ' >>> type (' kanji ') <type ' str ' >>> > Print binascii.b2a_hex (' kanji ') babad7d6>>> print binascii.b2a_hex (U ' kanji ') Traceback (most recent call last): File "<stdin>", line 1, in <module>unicodeencodeerror: ' ASCII ' codec can ' t encode characters inposition 0-1: o Rdinal not in range >>> print binascii.b2a_hex (U ' kanji '. Encode (' Utf-8 ')) e6b189e5ad97>>> print Binascii.b2a_hex (U ' kanji '. Encode (' GBK ')) babad7d6

Binascii is the binary conversion of data into ASCII, the above explanation is: ' Kanji ' type is str, binary is babad7d6,u ' kanji ' is unable to convert to ASCII, so that the first error is reported. The solution is to put it. Encode (' utf-8 ') into STR type. Because my command line is Windows default GBK encoding, all U ' Kanji ' .encode(‘gbk') , the output result is the same as the ' kanji ' result.

To summarize, Python's str is actually one of the Unicode, Python's default encoding is ASCII, for non-ASCII into ASCII will be error, keep in mind the following rules:

    1. Unicode = encode (' appropriate code ') = = str

    2. str = decode (' fit code ') = Unicode

There is also a simple way to set the code in the file header, you can save a lot of trouble:

Import sysreloads (SYS) sys.setdefaultencoding (' Utf-8 ')

For the second problem, it is the error when the file is read. Utf-8 file has a BOM and no BOM two ways, the difference seems to be in the BOM file than no BOM file more than a head, resulting in the utf-8 way to read the file times wrong, I have previously tried to read the document before the BOM to judge, skip the BOM file head, and then failed, really awkward ~ ~.

Also need Google to help God, the specific way is to use the codecs library to read the file (I guess this library is to detect the head of the file).

Import Codecscodecs.open (file_name, "R", encoding= ' utf-8 ', errors= ' ignore ')

For coding problems, be sure to understand how ASCII, Unicode, and utf-8 work.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.