"Reprint" has to know the Python string encoding related knowledge

Original address: http://www.cnblogs.com/Xjng/p/5093905.html

Development often encounters a variety of string encoding problems, such as error SyntaxError: Non-ASCII character ‘ascii‘ codec can‘t encode characters in position 0-2: ordinal not in range(128) , such as garbled display.
Because I did not know the principle of coding, encountered these situations, you can only continue to use a variety of coding decode and encode .....
Today, to tidy up a python of the various coding problems of the cause and solution, later encountered coding problems, will not like the rash of flies, everywhere crashed.

The following Python environment is 2.7, I heard that there is no coding problem in 3.X, because all the strings are Unicode, then a 3.X test.

If you do not know what is decode and encode, it is recommended to first look at: here

First, the role of encoding

1. In the python file, if there is Chinese, it is necessary to mark the first line of the file to use the type of encoding, for example #encoding=utf-8 , the use of UTF-8 encoding, what is the role of this encoding? What will change?
demo1.py

# encoding=utf-8test=‘测试test‘print type(test)print repr(test)

Output:

<type ‘str‘>‘\xe6\xb5\x8b\xe8\xaf\x95test‘

When we output a variable to the terminal through print, the IDE or the system will generally help us to convert the output, for example, the Chinese character will be converted into Mandarin, so the original content of the variable cannot be seen.
The REPR function can look at this variable in the form of Python, that is, to see the original content of this variable
From the above output can see the test variable str type, its encoding is utf-8 (how to know is Utf-8, see the third part), that is, the encoding type
If we change encoding to GBK,
demo2.py

# encoding=gbktest=‘测试test‘print type(test)print repr(test)

Output

<type ‘str‘>‘\xb2\xe2\xca\xd4test‘

This will change the encoding type of test to GBK.
So this encoding will decide how to encode the string variables defined in this PY file.
And if a variable is imported from another py file, or read from the database, Redis, and so on, what is its encoding?
a.py

# encoding=utf-8test=‘测试test‘

b.py

# encoding=gbkfrom a import testprint repr(test)

Output

‘\xe6\xb5\x8b\xe8\xaf\x95test‘

The test variable is defined in a.py, a.py is encoded by the utf-8,b.py encoding is gbk,b import test from A, and the result shows that test is still utf-8 encoded, which is a.py encoding
So encoding only determines the encoding of this py file, and does not affect the encoding of variables that are imported or read from elsewhere

Second, common error codec can‘t encode charactersThe cause

Python programs often get an error codec can‘t encode characters orcodec can‘t decode characters

Define a string in Python,

import sysprint sys.getdefaultencoding() # 输出 asciiunicode_test=u‘测试test‘print repr(str(unicode_test))

The above code will error

in position 0-1: ordinal not in range(128)

In addition to the Str method, if the operation of two has a Chinese string, will also error, but only one of the Chinese, but will not error

Unicode_test =U ' Test test%s{0} 'Print'%stest '% unicode_test# no ErrorPrint'%s test '% unicode_test#会报错Print Unicode_test%' Test ' #不会报错 print unicode_test%  ' test ' Span class= "hljs-comment" > #会报错 print Unicode_test.format ( Test ')  #不会报错 print Unicode_test.format (  ' test ')  #会报错 print unicode_test.split ( Test ')  #不会报错 print unicode_test.split (  ' test ')  #报错 print unicode_test +  ' test ' Span class= "hljs-comment" > #不会报错 print unicode_test +  ' test '  #会报错             

Why is that?
This is the reason for the following answer, here first listed the solution to the error:
The workaround is to set the system's default encoding to Utf-8

import sysreload(sys)sys.setdefaultencoding(‘utf-8‘)print sys.getdefaultencoding()unicode_test=u‘测试test‘

demo3.py
# Encoding=utf-8
Import Sys
Reload (SYS)
Sys.setdefaultencoding (' Utf-8 ')
Unicode_test=u ' testing test '
utf8_test= ' testing test '
Gbk_test=unicode_test.encode (' GBK ')

#合并unicode和utf-8Merge=unicode_test+utf8_testprint type (merge) print repr (merge) #合并unicode和gbk merge= Unicode_test+gbk_testprint type ( merge) print repr (merge) print merge# merge Utf-8 and Gbkmerge=utf8_test+ Gbk_testprint type (merge) Span class= "hljs-built_in" >print repr (merge) print merge            

This defines three strings that are unicode,utf-8 and GBK encoded respectively, Unicode_test,utf8_test and Gbk_test
1. When merging Unicode and Utf-8, the output:

‘unicode‘>u‘\u6d4b\u8bd5test\u6d4b\u8bd5test‘

The encoding of the merged result is Unicode encoding.
2. Merging Unicode and GBK will cause error:

‘utf8‘ codec can‘t decode byte 0xb2 in position 0: invalid start byte

So we can speculate that:
When Python operates on two strings, if the two strings have one Unicode encoding and one is non-Unicode encoded, Python decode the non-Unicode encoded string into Unicode encoding and then string manipulation
For example, the operation of merging strings can be written as the following function:

 def  Merge_strif isinstance (str1, Unicode) and not isinstance (STR2, Unicode): str2 = Str2.decode ( Sys.getdefaultencoding ()) elif not isinstance (str1, Unicode) and isinstance (STR2, Unicode): str1 = Str1.decode (sys.getdefaultencoding ()) return str1 + str2            

PS:sys.getdefaultencoding()的初始值是ascii
So
codec can‘t encode（decode） charactersThis error is generated by the two methods of encode or decode, and the parameter of this method is sys.getdefaultencoding (). If you use ASCII encoding to decode a string with Chinese characters, an error will be made. So modify the system's default code to avoid this error.
When the operation is performed, str Python executes unicode_test.encode(sys.getdefaultencoding()) and the error is also encountered.

3. #合并utf-8 and GBK do not error, Python will directly combine two strings, there will be no decode or encode operation, but the output, the part of the string will be garbled.
demo4.py

# encoding=gbkimport sysreload(sys)sys.setdefaultencoding(‘utf-8‘)unicode_test = u‘测试test‘utf8_test = unicode_test.encode(‘utf-8‘)gbk_test = unicode_test.encode(‘gbk‘)merge = utf8_test + gbk_testprint type(merge)print repr(merge)print merge

The file encoding here is gbk,sys.getdefaultencoding () set to Utf-8, the result is:

<type ‘str‘>‘\xe6\xb5\x8b\xe8\xaf\x95test\xb2\xe2\xca\xd4test‘测试test????test

That is, the GBK part garbled. So the output will be decoded according to the encoding of sys.getdefaultencoding ().

Third, how to determine the encoding of a string (string)

1. There is no way to accurately determine the encoding of a string, such as GBK "\aa" for A, utf-8 "\aa" for B, if given "\AA" how to determine what kind of code? It can be either a GBK or a utf-8
2. What we can do is to roughly determine the encoding of a string, because the above example of the situation is very few, more cases are GBK in the ' \aa ' represents a, utf-8 is garbled, such as, so we can determine that ' \aa ' is the GBK code, Because it doesn't make sense to use UTF-8 encoding to decode the results.

3. And the code that we often meet is basically only three kinds: utf-8,gbk,unicode

   • Unicode is typically the \u lead, followed by a four-digit number or string, for example \u6d4b\u8bd5 , one \u corresponding to a Chinese character
   • Utf-8 is generally the \x lead, followed by two letters or numbers, for example \xe6\xb5\x8b\xe8\xaf\x95\xe5\x95\x8a , three \x represents a Chinese character
   • GBK is generally the \x lead, followed by two letters or numbers, for example \xb2\xe2\xca\xd4\xb0\xa1 , two \x represent a Chinese character

4. Use the Chardet module to determine
   ```
   Import Chardet

Raw = U ' I'm a little bird '
Print Chardet.detect (Raw.encode (' Utf-8 '))
Print Chardet.detect (Raw.encode (' GBK '))
```
Output:

{‘confidence‘: 0.99, ‘encoding‘: ‘utf-8‘}{‘confidence‘: 0.99, ‘encoding‘: ‘GB2312‘}

The Chardet module calculates the probability that the string is an encoding, which is sufficient for a 99% application scenario.

Iv. String_escape and Unicode_escape1. String_escape

In Str, \x is a reserved character, indicating that the following two-bit character represents a word retranslated (so called, do not know right), for example ‘\xe6‘ , the general three characters retranslated represents a Chinese character
So when you define a variable, a=‘\xe6\x88\x91‘ it is the definition of a Chinese character "I", but sometimes we do not want a variable to represent a Chinese character, but instead to represent 3*4=12 characters, which can be used encode(‘string_escape‘) to convert:

‘\xe6\x88\x91‘.encode(‘string_escape‘)=‘\\xe6\\x88\\x91‘

Decode is the reverse.
The type before and after the conversion is string.
There is also a phenomenon, definition, a=‘\x‘ a=‘\x0‘ will be error ValueError: invalid \x escape , and the definition, that is, the backslash a=‘\a‘ is not followed by X, will be no problem, and the definition a=‘\x00‘ , that is, X followed by two characters, is no problem.

2. Unicode_escape

Similarly in Unicode, \u is a reserved character, which means that the next four characters represent a Chinese character, for example b=u‘\u6211‘ , "I:", and we want the B variable to represent 6 English characters instead of a Chinese character, you can use encode (' Unicode-escape ') to convert:

u‘\u6211‘.encode(‘unicode-escape‘)=‘\u6211‘

Note Encode is Unicode before it is converted to string.
In Unicode, \u is a reserved character, but in string it is not, so there is only one backslash, not two.
Decode is the reverse.
Likewise, a=‘\u‘ it will be an error.

3. Example

#正常的str和unicode字符str_char=' I ' Uni_char=U ' Me 'PrintRepr (Str_char)# ' \xe6\x88\x91 'PrintRepr (Uni_char)# u ' \u6211 '# decode (' Unicode-escape ') s1=' \u6211 ' s2=s1.decode (' Unicode-escape ')PrintRepr (S1)# ' \\u6211 'PrintREPR (S2)# u ' \u6211 '# encode (' Unicode-escape ') s1=U ' \u6211 ' s2=s1.encode (' Unicode-escape ')PrintRepr (S1)# u ' \u6211 'PrintREPR (S2)# ' \\u6211 '# decode ("String_escape") s1=‘\\Xe6\\x88\\x91 ' s2=s1.decode (' String_escape ')PrintRepr (S1)# ' \\xe6\\x88\\x91 '  Print repr (s2) # ' \xe6\x88\x91 ' < Span class= "hljs-comment" ># Encode ("String_escape") S1= \xe6\x88\x91 ' S2=s1.encode ( ' String_escape ') print repr (S1)  # ' \xe6\x88\x91 ' print  REPR (S2) # ' \\xe6\\x88\\x91 '       

4. Application

1. Content is Unicode, but type is str, you can use the decode("unicode_escape") convert to content and type are Unicode

   s1=‘\u6211‘s2=s1.decode(‘unicode-escape‘)

2. The content is str, but type is Unicode and can be used to encode("unicode_escape").decode("string_escape") convert to content and type are all str

   s1=u‘\xe6\x88\x91‘s2=s1.encode(‘unicode_escape‘).decode("string_escape")

Blog post for the author's original, without permission, no reprint.

"Reprint" has to know the Python string encoding related knowledge