Python Encoding Problems

After understanding the relationship between Unicode, UTF-8, and gb2312 encoding two days ago, I finally understood the Encoding Problems in Python. If we have a Chinese character string when writing a Python script, errors or garbled characters may occur during running. Generally, when #-*-coding: UTF-8-*-is added, no error will be reported, but there may be garbled characters, and the same code may have different results in different editors, have you ever encountered these problems and have no clue? I hope that my explanations will help you understand the problem.

A Python script has three types of codes:

1. script file encoding

The encoding format when the script is saved. Different Editors Use different formats, which may cause certain confusion. Generally, the encoding format is UTF-8 or cp936 (on Windows)

 

2. Python parser Encoding

Python uses ascci encoding by default. Therefore, if your script contains Chinese characters, parsing in the ascci Encoding Area will definitely fail, because ascci does not contain UTF-8 and cp936 contains Chinese characters. Therefore, you must add #-*-coding: UTF-8-*-annotation to the first line of the script to notify the python parser to use UTF-8, however, this is just to inform Python not to parse the Chinese character strings in the UTF-8 encoding area. For example, if you write a script and the code saved is cp936, an error will be reported at runtime if you do not add the annotation, because ascci does not include the characters in cp936, if you write the comments on the first line, the system will not report an error because UTF-8 contains this character, but the meaning of the character may not be that Chinese character. It is just a bit of processing. Note that if your script is encoded with UTF-8 and does not need to be annotated on the first line, python will correctly recognize it. I personally understand that python only recognizes UTF-8 and ANSI files. coding needs to be specified for ANSI format; otherwise, python uses ascci by default.

 

3. encoding of the environment where the script is run

If the encoding format of the runtime environment is the same as that of the script file, it will be correctly displayed. The difference is garbled because the runtime environment will be decoded.

 

Let's look at the Code:

#-*-Coding: UTF-8-*-# If this sentence does not exist, an error is returned. The default ascci cannot parse the cp936 encoding.

# Saving cp936 Encoding

 

STR = 'abc Chinese'

Print STR # correctly displayed in cmd, because CMD is cp936 encoded

USTR = u'abc Chinese' # An error is reported because cp936 cannot be decoded in UTF-8 format,

Ustr1 = Unicode (STR, 'cp936') # correct, because the encoding used for decoding is the same as that used for file storage.

 

========================================================== =====

# UTF-8 encoding for saving

 

STR = 'abc Chinese'

Print STR # cmd # No error is reported but it is garbled. If it is put into cygwin, the execution is correct because cygwin's bash is UTF-8

 

USTR = u'abc Chinese' # correct, because python uses the UTF-8 encoding set by default for UTF-8 encoded files

Print USTR # normal display, because Unicode is a "general" encoding set, it can be converted into other encoding methods

 

========================================================== ======

# Unicode encoding save, actually for utf-16le

STR = 'abc' # error, because the UTF-16 encoding will add two fffe bytes at the beginning, at this time python will use ascci by default for processing, because even if you add #-*-coding: UTF-16-*-is useless, because Python has not parsed this sentence has failed