The difference between str and Unicode in Python encoding is that when using python to process Chinese characters and reading files or messages, if garbled characters (string processing, read/write files, print) are found ), most people call encode/decode for debugging and do not have a clear idea about why garbled code occurs. today we will discuss how to handle the encoding problem.
Note: For Python2.x, it is not tested in Py3k.
Most common errors during debugging
Error 1
Traceback (most recent call last): File "
", line 1, in
UnicodeDecodeError: ‘ascii‘ codec can‘t decode byte 0xe6 in position 0: ordinal
Error 2
Traceback (most recent call last): File "
", line 1, in
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: ‘ascii‘ codec can‘t encode characters in position 0-1: ordinal not in range(128)
First
A general concept is required to understand character sets and character encoding.
ASCII | Unicode | UTF-8 | and so on
Character encoding notes: ASCII, Unicode and UTF-8
Str and unicode
Both str and unicode are subclasses of basestring.
So there is a way to determine whether it is a string
Def is_str (s): return isinstance (s, basestring)
Str and unicode conversion
Str-> decode ('The _ coding_of_str ')-> unicode-> encode ('The _ coding_you_want')-> str
Differences
Str is a byte string consisting of unicode encoded bytes.
Declaration method
>>> S = 'Chinese' s = u'chinese'. encode ('utf-8') >>> type ('Chinese ')
Length (number of returned bytes)
>>> U'chinese '. encode ('utf-8') '\ xe4 \ xb8 \ xad \ xe6 \ x96 \ x87'> len (u'chinese '. encode ('utf-8') 6
Unicode is a true string consisting of characters.
Declaration method
>>> S = u'chinese' >>> s = 'Chinese '. decode ('utf-8') >>> s = unicode ('Chinese', 'utf-8') >>> type (u'chinese ')
Evaluate the length (returns the number of characters ).
>>> U'chinese' u' \ u4e2d \ u6587 '>>> len (u'chinese') 2
Conclusion
Check whether str or unicode is to be processed. use the correct processing method (str. decode/unicode. encode)
The following method is used to determine whether it is unicode/str.
>>> Isinstance (u'chinese', unicode) True >>> isinstance ('Chinese', unicode) False >>> isinstance ('Chinese', str) true >>> isinstance (u 'Chinese', str) False
Simple principle: do not use encode for str, do not use decode for unicode (in fact, encode can be performed for str. for details, refer to the final section. it is not recommended to ensure simplicity)
>>> 'Chinese'. encode ('utf-8') Traceback (most recent call last): File"
", Line 1, in
UnicodeDecodeError: 'ascii 'codec can't decode byte 0xe4 in position 0: ordinal not in range (128) >>> u 'China '. decode ('utf-8') Traceback (most recent call last): File"
", Line 1, in
File "/System/Library/Frameworks/Python. framework/Versions/2.7/lib/python2.7/encodings/utf_8.py ", line 16, in decode return codecs. utf_8_decode (input, errors, True) UnicodeEncodeError: 'ascii 'codec can't encode characters in position 0-1: ordinal not in range (128)
Unicode is used as the intermediate encoding for different encoding conversions.
# S is code_A's str s. decode ('code _ A'). encode ('code _ B ')
File processing, IDE and console
You can use python as a pool, an entry, and an exit.
At the entrance, all are converted to unicode, and all are processed by unicode in the pool. at the exit, the data is converted to the target encoding (unless otherwise specified, the specific encoding is required in the processing logic)
Read file external input encoding, decode into unicode processing (internal encoding, unified unicode) encode into the required target encoding written to the target output (file or console)
Error reported by IDE and console, because the encoding is inconsistent with the IDE's own encoding during print
When the output is converted to the same encoding, the output can be normal.
>>> Print u 'Chinese'. encode ('gbk ')???? >>> Print u 'Chinese'. encode ('utf-8') Chinese
Suggestions
Standard code
Unified encoding to prevent garbled characters generated during a specific stage
Environment encoding, IDE/text editor, file encoding, database data table encoding
Ensure code source file encoding
This is important
The default encoding of the py file is ASCII. if non-ASCII characters are used in the source code file, the file header must be used for encoding declaration.
If it is not declared, an error occurs when the input is not ASCII. it must be placed in the first or second line of the file.
File "XXX.py", line 3 SyntaxError: Non-ASCII character ‘\xd6‘ in file c.py on line 3, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
Declaration method
#-*-Coding: UTF-8-*-or # coding = UTF-8
If the header declares coding = UTF-8, a = 'Chinese' is encoded as UTF-8
If the header declares coding = gb2312, a = 'Chinese' is encoded as gbk
So, all source file headers in the same project are encoded in a uniform way, and the declared encoding must be consistent with the encoding stored in the source file (editor-related)
Hard-coded strings used for processing in the source code, and unicode
Isolates its type from the encoding of the source file, so that it can be processed independently and without dependency
If s = u 'Chinese': # instead of s = 'Chinese' pass # note that when s is here, make sure it is converted to unicode
After completing the preceding steps, you only need to pay attention to the two unicode codes and the encoding you set (UTF-8 is generally used)
Processing sequence
1. Decode early 2. Unicode everywhere 3. Encode later
Related modules and methods
Obtain and set the system default encoding.
>>> import sys >>> sys.getdefaultencoding() ‘ascii‘ >>> reload(sys)
>>> sys.setdefaultencoding(‘utf-8‘) >>> sys.getdefaultencoding() ‘utf-8‘ >>> str.encode(‘other_coding‘)
In python, encode a specific encoding str into another encoding str.
# Str_A is UTF-8 str_A.encode ('gbk'). The operation is str_A.decode ('sys _ Codec '). encode ('gbk') here sys_codec is the sys. getdefaultencoding () encoding
'Get and set the system default encoding 'and 'str' here. the encode is related, but I rarely use it like this, mainly because I think it is complicated and uncontrollable, or I need to enter a clear decode and output a clear encode to make it easier.
Chardet
File encoding detection, download
>>> import chardet >>> f = open(‘test.txt‘,‘r‘) >>> result = chardet.detect(f.read()) >>> result {‘confidence‘: 0.99, ‘encoding‘: ‘utf-8‘}
\ U string to unicode string
>>> U' \ u4e2d> s = '\ u4e2d'> print s. decode ('unicode _ escape ') >>> a = '\ u4fee \ u6539 \ u8282 \ u70b9 \ u72b6 \ u6001 \ u6210 \ u529f' >>>. decode ('unicode _ escape ') u' \ u4133 \ u6539 \ u8282 \ u70b9 \ u72b6 \ u6001 \ u6210 \ u529f'