Python+requests Code identification bug

Source: Internet
Author: User

Requests is an HTTP library that uses Apache2 Licensed licenses. Written in Python, more user-friendly and easier to use.

Requests uses URLLIB3, so it inherits all of its features. Requests supports HTTP connection retention and connection pooling, supports the use of cookies to maintain sessions, supports file uploads, supports automatic encoding of response content, supports internationalized URLs and automatic POST data encoding. Modern, international, humane.

Recently in the process of using requests found a problem, is to grab some Chinese web pages, garbled, printing encoding is iso-8859-1. Why is that? By looking at the source code, I found that the default encoding recognition is relatively simple, directly from the response header file Content-type, if there is charset, it can be recognized correctly, if there is no charset but the text is considered to be iso-8859-1, See utils.py.


def get_encoding_from_headers (headers): "" "    Returns encodings from given HTTP Header Dict.    :p Aram Headers: Dictionary to extract encoding from.    "" " Content_Type = Headers.get (' content-type ')    if not content_type:        return None    content_type, params = Cgi.parse_header (content_type)    if ' charset ' in params:        return params[' CharSet '].strip ("' \" ")    if ' text ' In Content_Type:        return ' iso-8859-1 '

In fact requests offers to get the code from the content, just not used in the default, see utils.py:


def get_encodings_from_content (content): "" "    Returns encodings from given content string.    :p Aram Content: ByteString to extract encodings from.    "" " Charset_re = Re.compile (R ' <meta.*?charset=["\ ']* (. +?) ["\ ';] ', Flags=re. I)    pragma_re = Re.compile (R ' <meta.*?content=["\ ']*;? charset= (. +?) ["\ ';] ', Flags=re. I)    xml_re = Re.compile (R ' ^<\?xml.*?encoding=["\ ']* (. +?) return (content) +            pragma_re.findall (content) +            Xml_re.findall (content Charset_re.findall) ))



Encoding detection using Chardet is also available, see models.py:

@propertydef apparent_encoding: "" "the    apparent encoding, provided by the lovely Charade library    (Thanks, ian!).    "" " Return Chardet.detect (self.content) [' Encoding ']



How to fix this problem? Let's take a look at the example:

>>> r = requests.get (' http://cn.python-requests.org/en/latest/') >>> r.headers[' content-type '] ' Text/html ' >>> r.encoding ' iso-8859-1 ' >>> r.apparent_encoding ' utf-8 ' >>> Requests.utils.get_encodings_from_content (r.content) [' utf-8 ']>>> r = requests.get ('/HTTP/ Reader.360duzhe.com/2013_24/index.html ') >>> r.headers[' content-type '] ' text/html ' >>> r.encoding ' Iso-8859-1 ' >>> r.apparent_encoding ' gb2312 ' >>> requests.utils.get_encodings_from_content ( r.content) [' gb2312 ']



By understanding, you can solve this problem with a monkey patch:

Import Requestsdef Monkey_patch ():    prop = requests.models.Response.content    def content (self):        _content = Prop.fget (self)        if self.encoding = = ' iso-8859-1 ':            encodings = Requests.utils.get_encodings_from_content (_ Content)            If encodings:                self.encoding = encodings[0]            else:                self.encoding = self.apparent_encoding            _content = _content.decode (self.encoding, ' replace '). Encode (' UTF8 ', ' replace ')            self._content = _content        return _content    requests.models.Response.content = property (content) Monkey_patch ()

Related articles:

    • Requests:http for humans
    • Improvement scheme of grasping Chinese garbled characters in python+requests

Python+requests Code identification bug

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.