The Chinese coding problem of Python crawler

Python coding is a profound knowledge, and I am still bleeding python, so I am currently required to be only in their own crawl Web page to obtain Chinese information without error, only that, for other deeper content with the accumulation of knowledge presumably have a deeper understanding. The following is not my original understanding, but on the internet to read a lot of bloggers have ideas more intuitive expression before they can have a more direct understanding of these codes, thank them

The first is to attach the Chinese code comparison, more intuitive to show the effect of different coding on the output of text

The compilation environment is Wingide win8.1

Input #-*-coding:utf-8-*-
S= ' AB I'm a Chinese string '
Ss=u ' AB I'm a Chinese string '
U=s.decode (' Utf-8 ')
Print S
Print repr (s)
Print SS
Print REPR (ss)
Print U
Print repr (u)

Output

AB Contact Language Ionizing Juan Po Ying 楃 Juan? ' Ab\xe6\x88\x91\xe6\x98\xaf\xe4\xb8\xad\xe6\x96\x87\xe5\xad\x97\xe7\xac\xa6\xe4\xb8\xb2 '
AB I am a Chinese string
U ' ab\u6211\u662f\u4e2d\u6587\u5b57\u7b26\u4e32 '
AB I am a Chinese string
U ' ab\u6211\u662f\u4e2d\u6587\u5b57\u7b26\u4e32 '

From the above can be known when the input utf-8 format encoding, output text is garbled, and input Unicode when the output is not garbled

And decode is decoding, and decode (utf-8 (source page encoding format)) is the purpose of the original page encoding format into Unicode and then input, so that Chinese will not garbled

And encode is decoding, contrary to the above effect

Also introduce a chardet command, use this command can let you know the source page encoding is what, and then decode the good, the problem is solved perfectly.

The Chinese coding problem of Python crawler