Requests is an HTTP library that uses Apache2 Licensed licenses. Written in Python, more user-friendly and easier to use.
Requests uses URLLIB3, so it inherits all of its features. Requests supports HTTP connection retention and connection pooling, supports the use of cookies to maintain sessions, supports file uploads, supports automatic encoding of response content, supports internationalized URLs and automatic POST data encoding. Modern, international, humane.
Recently in the process of using requests found a problem, is to grab some Chinese web pages, garbled, printing encoding is iso-8859-1. Why is that? By looking at the source code, I found that the default encoding recognition is relatively simple, directly from the response header file Content-type, if there is charset, it can be recognized correctly, if there is no charset but the text is considered to be iso-8859-1, See utils.py.
def get_encoding_from_headers (headers): "" " Returns encodings from given HTTP Header Dict. :p Aram Headers: Dictionary to extract encoding from. "" " Content_Type = Headers.get (' content-type ') if not content_type: return None content_type, params = Cgi.parse_header (content_type) if ' charset ' in params: return params[' CharSet '].strip ("' \" ") if ' text ' In Content_Type: return ' iso-8859-1 '
In fact requests offers to get the code from the content, just not used in the default, see utils.py:
def get_encodings_from_content (content): "" " Returns encodings from given content string. :p Aram Content: ByteString to extract encodings from. "" " Charset_re = Re.compile (R ' <meta.*?charset=["\ ']* (. +?) ["\ ';] ', Flags=re. I) pragma_re = Re.compile (R ' <meta.*?content=["\ ']*;? charset= (. +?) ["\ ';] ', Flags=re. I) xml_re = Re.compile (R ' ^<\?xml.*?encoding=["\ ']* (. +?) return (content) + pragma_re.findall (content) + Xml_re.findall (content Charset_re.findall) ))
Encoding detection using Chardet is also available, see models.py:
@propertydef apparent_encoding: "" "the apparent encoding, provided by the lovely Charade library (Thanks, ian!). "" " Return Chardet.detect (self.content) [' Encoding ']
How to fix this problem? Let's take a look at the example:
>>> r = requests.get (' http://cn.python-requests.org/en/latest/') >>> r.headers[' content-type '] ' Text/html ' >>> r.encoding ' iso-8859-1 ' >>> r.apparent_encoding ' utf-8 ' >>> Requests.utils.get_encodings_from_content (r.content) [' utf-8 ']>>> r = requests.get ('/HTTP/ Reader.360duzhe.com/2013_24/index.html ') >>> r.headers[' content-type '] ' text/html ' >>> r.encoding ' Iso-8859-1 ' >>> r.apparent_encoding ' gb2312 ' >>> requests.utils.get_encodings_from_content ( r.content) [' gb2312 ']
By understanding, you can solve this problem with a monkey patch:
Import Requestsdef Monkey_patch (): prop = requests.models.Response.content def content (self): _content = Prop.fget (self) if self.encoding = = ' iso-8859-1 ': encodings = Requests.utils.get_encodings_from_content (_ Content) If encodings: self.encoding = encodings[0] else: self.encoding = self.apparent_encoding _content = _content.decode (self.encoding, ' replace '). Encode (' UTF8 ', ' replace ') self._content = _content return _content requests.models.Response.content = property (content) Monkey_patch ()
Related articles:
- Requests:http for humans
- Improvement scheme of grasping Chinese garbled characters in python+requests
Python+requests Code identification bug