Python Open source Project scrapy crawl file garbled solution

Scrapy the page to grasp the time, the saved file appears garbled, after analysis is the reason for encoding, only need to convert the code to Utf-8 can, snippets

.... import Chardet .....
Content_Type = Chardet.detect (html_content) #print (content_type[' encoding ')) ifcontent_type[' encoding ']! = "UTF       -8 ": Html_content =html_content.decode (content_type[' encoding ') html_content = Html_content.encode (" Utf-8 ") Open (filename, "WB"). Write (Html_content) .... This saves the file is Chinese. Step: First convert the gb2312 encoding to Unicode encoding and then convert the Unicode encoding to Utf-8.

Python Open source Project scrapy crawl file garbled solution