Python2.7.9 Coding Problems

Recently learned a network crawler, encountered the first headache is the coding problem, read a lot of tutorials are not clear, now tidy up, I hope to see more convenient later

Using sys.getdefaultencoding () to view Python's default system encoding, the output is ASCII, this encoding should refer to the source file encoding format, can be saved as a way to change to Utf-8, but in order to avoid each file is saved as, you can To be in the ... Create a new file in the Python27\lib\site-packages directory sitecustomize.py, add the following lines, set the default encoding to UTF-8 format

# Encoding=utf-8 Import sysreload (SYS) sys.setdefaultencoding ('UTF8')

This time opening the Python Shell with sys.getdefaultencoding () to check the encoding format will get Utf-8

Use  content = Urllib2.urlopen (URL)    open a Web page and get the content of the Web page. If you want to output this content string, you will get an error:   [decode error-output not utf-8]   in Python, string types can be viewed using type (), which is divided into two types of STR and Unicode

 >>> s =  "  Chinese   '  >>> ; 

Where the str type is utf-8 under the Linux system, the string should be encoded as utf-8 before the output, and if it is GBK encoded, retVal = Retval.decode ('gbk ') should be used. '). Encode ('UTF8') is converted to utf-8 format for normal output. The default encoding under Windows is GBK encoding, so it must be GBK encoded before the output.

Sometimes crawl a Web page, in the meta and header does not describe the encoding format, you can use the chardet module to detect the file encoding format

Import= open ('save.html','r'=  Chardet.detect (F.read ())print fencoding

The above code output style is: {'confidence': 0.99, 'encoding' : 'utf-8' , you can know that the file encoding format for Utf-8 is open.

　　

Python2.7.9 Coding Problems