Troubleshooting Python Requests Library encoding SOCKS5 Agent

This article mainly introduces the solution to the Python requests Library coding SOCKS5 Agent problem, has a certain reference value, now share to everyone, the need for friends can refer to

Coding issues

Response = Requests.get (URL, Params=params,      headers=headers, timeout=10) print ' self.encoding ', Response.encodingoutput:self.encoding iso-8859-1

Check some relevant information, see the next requests source, only in the server response to the head contains Content-type, and there is charset information, requests can correctly identify, otherwise will use the default iso-8859-1 encoding. This is also discussed on GitHub, but requests's authors say they are based on the RFC.

In the above code, Response.text is the Unicode encoded content of the requests library return response

In this way, when we go to the response content of some Chinese web pages, and the response header does not have charset information, then Response.text encoding will be problematic (requests JSON () method is also affected by this encoding)

For example, when I crawl Baidu's Web page, where the text is UTF-8 encoded

The following python2.7 code

in [+]: a = ' approx ' #utf-8 encoded in [[]: aout[15]: ' \xe7\xba\xa6 ' in []: B=a.decode (' iso-8859-1 ') #response. Text that the response content is ISO-8859 -1 encoding, decode it to Unicodein []: bout[23]: U ' \xe7\xba\xa6 ' in []: C=b.encode (' UTF8 ') #如果我们没有注意ISO-8859-1, Encode it directly with UTF8 in [the]: cout[27]: ' \xc3\xa7\xc2\xba\xc2\xa6 ' #那么encode得到的utf-8, on the display is garbled, because ' about ' the Utf-8 encoding is ' \xe7\xba\ Xa6 '

Solution 1: use response.content, response.content in bytes, so you can use content to determine its own code

Workaround 2: Use response.encoding = ' utf-8 ' after obtaining the request

Workaround 3: Use the Requests library to determine the function of the encoding according to the content of the response, as mentioned in the reference literature

Python2 's code is still messy. STR can be various encodings, Python3 Unified Str is Unicode, byte can be a variety of encodings

Python2 is encode after the STR type, after decode is the Unicode type, Python3 encode is the byte type, after Decode is the STR type (Unicode encoding)

With Python3, here's the code for Python3.

in [+]: a = ' approx ' #UnicodeIn [+]: Type (a) out[14]: Strin []: B=a.encode (' UTF8 ') in [+]: bout[16]: B ' \xe7\xba\xa6 ' in [17]: Type (b) out[17]: Bytesin [+]: B ' \xe7\xba\xa623,000 '. Decode (' iso-8859-1 ') out[27]: ' 约23,000 ' in [+]: type (b ' \xe7\xba \xa623,000 '. Decode (' iso-8859-1 ')) out[28]: Strin []: B ' \xe7\xba\xa623,000 '. Decode (' UTF8 ') out[29]: ' About 23,000 '

SOCKS5 Agent Issues

Now the requests2.13.0 of the SOCKS5 agent I use when the problem will arise,

The agent I'm using is shadowsocks, for example, I want to access https://www.facebook.com when I send a SOCKS5 request to a local 127.0.0.1:1080 port, I find that shadowsocks is connected to an IP address and not connected , I used Chrome to connect to Facebook, I found that shadowsocks is connected to the www.facebook.com, can successfully connect, it should be a DNS resolution problem, there is a problem of repeated resolution, using requests2.12 will not have this problem, Also found the relevant issue on GitHub.

Import requestsheaders = {' user-agent ': ' mozilla/5.0 (X11; Linux x86_64) '   applewebkit/537.36 (khtml, like Gecko) '   chrome/56.0.2924.87 safari/537.36 '}proxies = {' http ' : ' socks5://127.0.0.1:1080 ', ' https ': ' socks5://127.0.0.1:1080 '}url = ' https://www.facebook.com ' response = Requests.get (URL, proxies=proxies) print (response.content)