Python Crawler series: Several methods to judge the target page encoding

When crawling page content, it is important to understand the encoding used by the target site, and this article describes several common methods and uses several sites for simple testing.

Code Run Result:

From the test results of several websites in different countries, UTF8 use more (for pure English site, in what way decoding no essential difference). However, the success rate of the method 3 and 4 is higher, but the speed is slightly slower, the better method is the combination of these methods, combined with the exception handling structure, priority to use Method 1 or 2, it is not possible to use the latter method. Http://www.aibbt.com/a/18106.html

Collecting and hiding

Python Crawler series: Several methods to judge the target page encoding