Python crawls and saves html pages with garbled characters.

Python crawls and saves html pages with garbled characters.

This example describes how to solve the garbled problem when python crawls and saves html pages. We will share this with you for your reference. The details are as follows:

When crawling html pages and saving them with Python, the contents of captured webpages are often garbled. The cause of this problem is that the encoding settings in your code are incorrect. On the other hand, the actual encoding and the labeled encoding of the webpage are incorrect when the encoding settings are correct. The encoding of the html page is as follows:
Copy codeThe Code is as follows: <meta http-equiv = "Content-Type" content = "text/html; charset = UTF-8"/>

Here is a simple solution: Use chardet to determine the real code of the webpage, and determine the flag encoding from the info returned by the url request. If the two encoding types are different, the bs module is extended to GB18030 encoding. If the two encoding types are the same, files are written directly (the default encoding is UTF-8 here ).

import urllib2import sysimport bs4import chardetreload(sys)sys.setdefaultencoding('utf-8')def download(url):  htmlfile = open('test.html','w')  try:    result = urllib2.urlopen(url)    content = result.read()    info = result.info()    result.close()  except Exception,e:    print 'download error!!!'    print e  else:    if content != None:      charset1 = (chardet.detect(content))['encoding'] #real encoding type      charset2 = info.getparam('charset') #declared encoding type      print charset1,' ', charset2      # case1: charset is not None.      if charset1 != None and charset2 != None and charset1.lower() != charset2.lower():        newcont = bs4.BeautifulSoup(content, from_encoding='GB18030')  #coding: GB18030        for cont in newcont:          htmlfile.write('%s\n'%cont)      # case2: either charset is None, or charset is the same.      else:        #print sys.getdefaultencoding()        htmlfile.write(content) #default coding: utf-8  htmlfile.close()if __name__ == "__main__":  url = 'http://www.bkjia.com'  download(url)

Test.html file to open the following, you can see that the use of UTF-8 without BOM encoding format storage, that is, we set the default encoding: