Python coding (1): python Coding

Reprinted: python encoding (1), python Encoding

The following content is reproduced from:

Http://in355hz.iteye.com/blog/1860787

 

Recently, some scripts need to be written in Python. Although the script interaction is only command line + log output, I decided to output log information in Chinese to make the interface more friendly.

Soon, I encountered an exception:

UnicodeEncodeError: 'ascii 'codec can't encode characters in position 0-3: ordinal notin range (128)

To solve the problem, I took the time to study the character encoding processing in Python. Many articles on the internet talk about Python character encoding, but I have read it once and I think I can better understand it.

Next, let's repeat the basics of the Python string. If you are familiar with this content, you can skip it.

 

Corresponding to the char and wchar_t of C/C ++, Python also has two types of strings, str and unicode:

#-*-Coding: UTF-8-*-# file: example1.pyimport string # This is str's string s = ' '# This is a unicode string u = u' 'print isinstance (s, str) # Trueprint isinstance (u, unicode) # Trueprint s. _ class _ # <type 'str'> print u. _ class _ # <type 'unicode '>

Previous statement:#-*-Coding: UTF-8 -*-It indicates that the above Python code is UTF-8 encoded.

In order to ensure that the output does not show garbled characters on the linux terminal, you need to set the linux environment variable: export LANG = en_US.UTF-8

If you use SecureCRT like me, set Session Options/Terminal/Appearance/Character Encoding to a UTF-8 to ensure that the output of the linux Terminal is correctly decoded.

You can use the encode/decode method to convert two Python string types:

# Converting from str to unicodeprint s. decode ('utf-8') # # converting from unicode to strprint u. encode ('utf-8') #

Why is the conversion from unicode to str is encode, and the reverse is decode?

Because Python thinks that 16-bit unicode is the only internal character code, and common character sets such as gb2312, gb18030/gbk, UTF-8, and ascii are character binary (byte) encoding. To convert a character from unicode to binary encoding, encode is required.

In Python, str is an ansi string encoded by character set. Python itself does not know the str encoding. Developers must specify the correct character set decode.

(In addition, Python knows str encoding. Because we stated before the code#-*-Coding: UTF-8 -*-This indicates that str in the Code is UTF-8 encoded. I don't know why Python does not .)

What if I use the wrong character set for encode/decode?

 

# Use ascii encoding to encode a unicode string u containing Chinese characters. encode ('ascii ') # error, because Chinese characters cannot be encoded using the ascii character set # UnicodeEncodeError: 'ascii 'codec can' t encode characters in position 0-3: ordinal not in range (128) # Use gbk to encode the unicode string u containing Chinese characters. encode ('gbk') # correct, because the 'Customs Inspector 'can be expressed in the Chinese gbk character set #' \ xb9 \ xd8 \ xb9 \ xd8 \ xf6 \ xc2 \ xf0 \ xaf' # The str above the direct print will display garbled characters, modify the environment variable to zh_CN.GBK. The result is correct. # use ascii to decode the UTF-8 string s. decode ('ascii ') # error. Chinese UTF-8 characters cannot be decoded using ascii # UnicodeDecodeError: 'ascii 'codec can' t decode byte 0xe5 in position 0: ordinal not in range (128) # Use gbk to decode UTF-8 string s. decode ('gbk') # No error, but the result of decoding the UTF-8 encoding stream with gbk is obviously only garbled # U' \ u934f \ u51b2 \ u53e7 \ u95c6 \ u5ea8 \ u7b2d'

 

This encountered the exception I posted at the beginning of this article: UnicodeEncodeError: 'ascii 'codec can't encode characters in position 0-3: ordinal not in range (128)

Now we know that this is a string encoding exception. Next, why is a string encoding/decoding exception so easy in Python?

This should mention two traps that are easy to encounter when processing Python encoding. The first is about string connection:

 

#-*-Coding: UTF-8-*-# file: example2.py # This is the str string s = ' '# This is the unicode string u = u' s's + u # failure, UnicodeDecodeError: 'ascii 'codec can't decode byte 0xe5 in position 0: ordinal not in range (128)

Will a decoding error occur for a simple string connection?

Trap 1: during operations that contain both str and unicode, Python converts str to unicode before the operation. Of course, the operation results are also unicode.

Because Python does not know the str encoding beforehand, it can only use sys. getdefaultencoding () encoding to decode. In my impression, the value of sys. getdefaultencoding () is always 'ascii '-- obviously, if the str to be converted has Chinese characters, an error will certainly occur.

Except for string connections, the % operation results are the same:

# Correct. All strings are str and decode is not required. "Chinese: % s" % s # Chinese: # failed, equivalent to running: "Chinese: % s ". decode ('ascii ') % u "Chinese: % s" % u # UnicodeDecodeError: 'ascii 'codec can' t decode byte 0xe5 in position 0: ordinal not in range (128) # correct. All strings are unicode and do not require decodeu "Chinese: % s" % u # Chinese: # failure, equivalent to running: u "Chinese: % s" % s. decode ('ascii ') u "Chinese: % s" % s # UnicodeDecodeError: 'ascii 'codec can' t decode byte 0xe5 in position 0: ordinal not in range (128)

I don't understand why sys. getdefaultencoding () has nothing to do with the environment variable $ LANG. If Python uses $ LANG to set the sys. getdefaultencoding () value, at least the developer's chance of encountering UnicodeDecodeError is reduced by 50%.

In addition, as mentioned above, I also doubt why Python is not referenced here.#-*-Coding: UTF-8 -*-Because Python always checks your code before running, which ensures that the str defined in the Code must be UTF-8.

To solve this problem, I only recommend that you write u before the Chinese character string in the code. In addition, str has been canceled in Python 3 so that all strings are unicode-this may be the correct decision.

In fact, the value of sys. getdefaultencoding () can be modified using the "backdoor" method. I do not particularly recommend this solution, but I will post it as it is useful in the future:

#-*-Coding: UTF-8-*-# file: example3.pyimport sys # This is the str string s = ' '# This is the unicode string u = u'' # Make sys. the value of getdefaultencoding () is 'utf-8' reload (sys) # reload to call the setdefaultencoding method sys. setdefaultencoding ('utf-8 ') # Set 'utf-8' # No problem s + u # U' \ u5173 \ u5173 \ u96ce \ u9e20 \ u5173 \ u5173 \ u96ce \ u9e20' # No problem "Chinese: % s "% u # U' \ u4e2d \ u6587 \ uff1a \ u5173 \ u5173 \ u96ce \ u9e20' # Still no problem u" Chinese: % s "% s # U' \ u4e2d \ u6587 \ uff1a \ u5173 \ u5173 \ u96ce \ u9e20'

We can see that the problem is solved by magic. But note! Sys. setdefaultencoding.

Another trap is related to standard output.

What happened just now? I always said that you should set the correct linux $ LANG environment variable. So what if I set the wrong $ LANG, such as zh_CN.GBK? (To avoid terminal impact, set SecureCRT to the same character set .)

It is obviously garbled, but not all outputs are garbled.

 

#-*-Coding: UTF-8-*-# file: example4.pyimport string # This is the string s of str = ' '# This is the unicode string u = u'' # output the str string, the display is garbled print s # The output unicode string is correct print u #

 

Why is unicode rather than str correctly displayed? First, we need to understand print. Like all languages, this Python Command actually prints characters to the standard output stream-sys. stdout. Python has changed its magic here. It will encode unicode according to sys. stdout. encoding, and directly output str to the operating system.

This is also why we need to set the linux $ LANG environment variable to be consistent with the SecureCRT. Otherwise, these characters will be converted again by SecureCRT before being displayed to the desktop Windows System Using the encoding CP936 or GBK.

In general, the value of sys. stdout. encoding is consistent with that of the linux $ LANG environment variable:

 

#-*-Coding: UTF-8-*-# file: example5.pyimport sys # Check the encoding of the standard output stream print sys. stdout. encoding # Set $ LANG = zh_CN.GBK, output GBK # Set $ LANG = en_US.UTF-8, output UTF-8 # This is the unicode string u = u 'Customs '# output unicode string, print u is displayed correctly #

 

However, there is a trap 2: Once your Python code runs in the pipeline/sub-process mode, sys. stdout. encoding will become invalid and you will encounter UnicodeEncodeError again.

For example, run the above example4.py code in MPs queue mode:

python -u example5.py | moreUnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)None

 

First, the value of sys. stdout. encoding is set to None. Second, Python tries to encode unicode with ascii during print.

Because the ascii character set cannot be used to represent Chinese characters, the encoding fails.

How can this problem be solved? I don't know how others do it. In short, I used an ugly method:

#-*-Coding: UTF-8-*-# file: example6.pyimport osimport sysimport codecs # In any case, use the current character set in linux to output: if sys. stdout. encoding is None: enc = OS. environ ['lang ']. split ('. ') [1] sys. stdout = codecs. getwriter (enc) (sys. stdout) # Replace sys. stdout # This is the unicode string u = u' '# output unicode string, correct print u #

 

This method still has a side effect: directly outputting the Chinese str will fail, because the writer and sys. stdout uses sys. the character set of getdefaultencoding () is converted to unicode output.

# This is the str string s = ' '# output str string, exception print s # UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range (128)

Apparently, the value of sys. getdefaultencoding () is 'ascii 'and the encoding fails.

Solution: As mentioned in example3.py, you can either declare u to unicode for str, or modify sys. getdefaultencoding () through a "backdoor ():

# Make sys. the value of getdefaultencoding () is 'utf-8' reload (sys) # reload to call the setdefaultencoding method sys. setdefaultencoding ('utf-8') # Set 'utf-8' # This is the str string s = ' '# output the str string, OKprint s #

All in all, Chinese Input and Output in Python 2 is a matter of crisis, especially when str and unicode are mixed in your code.

Some modules, such as json, will directly return unicode-type strings, so that your % operation requires character decoding and failure. Some will directly return str, and you need to know their real encoding, especially in print.

To avoid some traps, as mentioned above, the best way is to always use u to define Chinese strings in Python code. In addition, if your code needs to run in the pipeline/sub-process mode, you need the skills in example6.py.

 

(End)