Compilation of coding knowledge in Python

problem

In the usual work, encountered such a mistake:

Unicodedecodeerror: ' ASCII ' codec can ' t decode byte

Presumably everyone has met, very common. So I decided to do a collation and study of Python coding.

Basic knowledge

In python2.x, there are two types of data, Unicode and STR, both of which are subclasses of basestring

>>> a = ' Medium ' >>> type (a)
 
  
   
  >>> isinstance (a,basestring) true>>> a = U ' >>> type (a)
  
   
    
   >>> isinstance (a,basestring) True
  
   
 
  

The difference, in a nutshell, is that STR is a string of bytes, made up of Unicode encoded (encode) bytes (like a byte with python3.x); Unicode is an object, which is a literal character.

>>> a= ' Chinese ' >>> len (a) 6>>> repr (a) "' \\xe4\\xb8\\xad\\xe6\\x96\\x87 '" >>> b=u ' Chinese ' >>> Len (b) 2>>> repr (b) "U ' \\u4e2d\\u6587 '"

Consoles and scripts

Under Linux, the Python console executes the following commands, resulting in a different result from the execution script

>>> a = U ' Chinese ' >>> repr (a) "U ' \\xe4\\xb8\\xad\\xe6\\x96\\x87 '" >>> b = Unicode (' Chinese ', ' utf-8 ') b) >>> repr (b) "U ' \\u4e2d\\u6587 '"

You can see that the U ' Chinese ' initialized object A is not what we expected, what is the reason?
Consider python as a pipe in which the intermediate processes in the pipe are processed using Unicode. At the entrance, turn all the way to Unicode, exit, and then to the target code (with the exception of course, the case where the specific encoding is used in the processing logic).
In the console execute command a = U ' Chinese ', which can be interpreted as a command, a = ' Chinese '. Decode (encode), thereby to Unicode object A. So what's the encode here? For the console, it is the standard input, i.e. sys.stdin.encoding

>>> sys.stdin.encoding ' iso-8859-1 '

My side of the console. The default encoding is iso-8859-1, so a = U ' chinese ' <=> a = ' Chinese '. Decode (' iso-8859-1 ')
Here the ' Chinese ' is understood by the console, even after encoding the bytecode according to the terminal code, for UTF-8 encoded terminal, ' Chinese ' = ' \\xe4\\xb8\\xad\\xe6\\x96\\x87 '

>>> a= ' Chinese ' decode (' iso-8859-1 ') >>> repr (a) "U ' \\xe4\\xb8\\xad\\xe6\\x96\\x87 '"

So how do you modify this coded value and set it to what? In the Linux environment, set the environment variable method as follows, the specific setting what as long as the terminal encoding method has been

Export Pythonioencoding=utf-8

Summarize

Back to the original problem, the problem was caused by the lack of clarity between Unicode and STR, and the mixing of the two.

>>> a = ' Chinese ' >>> a.encode (' GBK ') Traceback (most recent call last): File "
 
  
   
  ", line 1, in 
  
   
    
   unicodedecodeerror: ' ASCII ' codec can ' t decode byte 0xe4 in position 0:ordinal
  
   No in range
 /c2>
  

The above object A is actually STR, that is, byte code, if the terminal is Utf-8 encoding, then A is used Utf-8 encode byte code. A.encode (' GBK ') is equivalent to A.decode (encoding). Encode (' GBK '), which first decodes the bytecode into Unicode characters and then encode to bytecode. The Unicode object acts as a broker. So what's the encoding here?

>>> import sys>>> sys.getdefaultencoding () ' ASCII '

The default is ASCII, which is why the error cannot be decoded with ASCII

>>> Reload (SYS)
 
  
   
  >>> sys.setdefaultencoding (' Utf-8 ') >>> a = ' Chinese ' >>> Repr (a) "' \\xe4\\xb8\\xad\\xe6\\x96\\x87 '" >>> a.encode (' GBK ') ' \xd6\xd0\xce\xc4 '
 
  

Change the default encoding to Utf-8. The use of the Encode method for STR is discouraged because STR is implicitly decoded. Decode only for Str,encode Unicode only, all Decode/encode displays the specified encoding method.

