First, these problems only occur in python2.X, because in python 3. X, only unicode strings are available, that is, all processed in the program are automatically converted to unicode strings. So how can we avoid and clarify the encoding problem during python Program Development in 2.x? First of all, we must maintain a good unified rule. Otherwise, everything will be white. It is best to use UTF-8 in a unified manner.
1.Process non-ASCIIEncoding
Python uses ascii encoding by default. When processing non-ASCII encoding in python, the following error occurs frequently:
UnicodeDecodeError: 'ascii 'codec can't decode byte 0x ?? In position 1: ordinal not in range (128)
0x ?? Is a number that exceeds 128
We often add the encoding attribute at the beginning of the file: #-*-coding = utf8 -*-
Therefore, if you cannot process other encodings, you need to set the python default encoding to the required encoding. There are two main methods:
01.Preferred method
Import sys reload (sys) # reload sys
Sys. setdefaultencoding ('utf-8') # Check whether your encoding requires UTF-8 or gb2312.
Why do we have to reload the sys module once before calling setdefaultencoding? Because the import statement here is not the first import Statement of sys, that is to say, it may be the second or third import of the sys module. Here it is just a reference to sys, only reload can be used for re-loading. Why is it necessary to re-load the function, but it cannot be called if it is directly referenced? Because the setdefaultencoding function is deleted after it is called by the system, it is no longer available when it is referenced through import. Therefore, the sys module must be reloaded once so that setdefaultencoding can be available, in order to modify the interpreter's current character encoding in the code.
02.Method2-Global settings
Create a sitecustomize. py file in the Lib \ site-packages folder of Python (Sitecustomize. pyIs a special file, Python will try to load the file at startup, so all code will run the file), you can automatically set the code.
Import sys
Sys. setdefaultencoding ('gb2312 ')
3.Check the current Encoding
Import sys
Sys. getdefaultencoding ()
2.Character encoding judgment
Chardet can be used to detect the encoding of strings/files.
01. chardetInstallation
You can use the easy_install tool to quickly install chardet. The command is as follows: easy_install.exe chardet
02. chardetUse
Chardet can directly use the detect function to detect the encoding of the given character. The Return Value of the function is a dictionary with two elements. One is the credibility of the detection, and the other is the detected encoding.
Import urllib
Import chardet
Rawdata = urllib. urlopen ('HTTP: // www.sina.com.cn/'). read ()
Print chardet. detect (rawdata)
# Result: {'credentials': 0.99, 'encoding': 'gb2312 '}
3.Decoding of File Processing
Response = urllib. urlopen (url)
Text = response. read (). decode ("UTF-8") # add by insun
According to step 1, we set utf8 encoding and wrote a program to capture Douban mp3. The mp3 names stored in the program are garbled and printed out. The missing mp3 files are all correct Chinese characters.
When there are already too many threads, there are already too many threads
At this time, it is obvious that decoding is required.
Decode ("UTF-8 ")
Let's not go further without the BOM header issue.
4. PythonOperationMysqlChinese garbled characters
To operate on MySQL using Python, you must install Python-MySQL.
You can search for it from the Internet and install it like a general Python package.
After installation, the module name is MySQLdb, which can be used in Windows and Linux environments.
Use the following measures to ensure that MySQL output is not messy:
1 PythonFile EncodingUTF-8(Add# Encoding = UTF-8)
2 MySQLDatabaseCharset = UTF-8
3 PythonConnectionMySQLAdd ParametersCharset = utf8
4SetPythonThe default encoding isUTF-8 (sys. setdefaultencoding (UTF-8)
# Encoding = UTF-8
Import sysimport MySQLdb reload (sys) sys. setdefaultencoding ('utf-8') Export cute ('use mydb'export cur.exe cute ('select * from mytb limit 100') f = file ("/home/user/work/tem.txt ", 'W') for I in cur. fetchall (): f. write (str (I) f. write ("") f. close () cur. close ()