Python character set conversion problem handling scheme sharing when crawling Web pages

Questions raised:

Sometimes we collect the Web page, after processing the string to save to file or write to the database, this time need to develop a string encoding, if the collection page encoding is gb2312, and our database is utf-8, so do not do any processing directly into the database may be garbled (not tested, Do not know whether the database will be automatically transcoded), we need to manually convert gb2312 to Utf-8.

First of all we know that the characters in Python are ASCII code by default, English of course, no problem, when the Chinese language immediately to kneel.

I don't know if you remember, Python, when you print Chinese characters, you need to precede the string with U:

Print U "to get a base? "

In this way the Chinese can show that the role of u in this is to convert the following string into Unicode code, so that Chinese can be displayed correctly.
There is a Unicode () function associated with it, which is used as follows

Str= "Come on base" Str=unicode (str, "Utf-8") print str

The difference between you and U is that it uses Unicode to convert STR to Unicode encoding, the second parameter needs to be correctly specified, and Utf-8 is the file character set of my test.py script itself, which is probably ANSI by default.
Unicode This is a key that continues below

We started to crawl Baidu home page, note, visitors visit Baidu home page, view the source code, it's charset=gb2312.

Import Urllib2def Main ():  f=urllib2.urlopen ("http://www.baidu.com")  str=f.read ()  str=unicode (str, " gb2312 ")  fp=open (" baidu.html "," W ")  Fp.write (Str.encode (" Utf-8 "))  fp.close () if __name__ = = ' __main__ ':  Main ()

Explain:
We first use the Urllib2.urlopen () method to crawl the Baidu homepage, F is the handle, with Str=f.read () to read all the source code into STR

Understand, str inside is we crawl HTML source code, because the page default character set is gb2312, so if we save directly to the file, the file encoding will be ANSI.

For most people, this is enough, but sometimes I want to convert gb2312 to Utf-8.

First of all:
Str=unicode (str, "gb2312") #这里的gb2312就是str的实际字符集, we now convert it to Unicode

And then:
Str=str.encode ("Utf-8") #将unicode的字符串重新编码成utf-8

At last:

Write str to the file, open the file to look at the encoding properties, found that it is utf-8, the utf-8 transcoding.

Summarize:

We recall that if you need to save the string in the specified character set, there are several steps:

1: Decoding str to Unicode string using Unicode (str, "original encoding")

2: Convert Unicode string str using Str.encode ("specified character set") to the character set you specified

3: Save str file, or write to the database and so on, of course, the code you have specified, is not it?

