Python resists anti-crawler by disguising head data

This article mainly describes the python through the camouflage head data resistance anti-Crawler, has a certain reference value, now share to everyone, the need for friends can refer to

0x00 Environment

System Environment: WIN10

Authoring tools: JetBrains pycharm Community Edition 2017.1.2 x64

Python version: python-3.6.2

Grab Bag tool: Fiddler 4

0X01 Head Data Camouflage idea

Data is submitted to the server via HTTP, and the following is a fiddler crawl of Python's non-spoofed message header information

get/u012870721 http/1.1accept-encoding:identityhost:blog.csdn.netuser-agent: <span style= "color: #ff0000;" >python-urllib/3.6</span>connection:close

python-urllib/3.6

Obviously, we were exposed. Now it's time to ask, how! Simulate the browser, to disguise itself as a browser, the browser to access the header data sent

connection:keep-aliveupgrade-insecure-requests:1user-agent:mozilla/5.0 (Windows NT 10.0; Win64; x64) applewebkit/537.36 (khtml, like Gecko) chrome/60.0.3112.113 safari/537.36referer:http://write.blog.csdn.net/ Postlistaccept-encoding:gzip, deflateaccept-language:zh-cn,zh;q=0.8

0x02 Code Implementation

From urllib Import requesthtml_url = "http://blog.csdn.net/u012870721"; #伪装构造头header ={"Connection": "Keep-alive", " Upgrade-insecure-requests ":" 1 "," user-agent ":" mozilla/5.0 (Windows NT 10.0; Win64; x64) applewebkit/537.36 (khtml, like Gecko) chrome/60.0.3112.113 safari/537.36 "," Accept ":" text/html,application/ xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 "," accept-encoding ":" Gzip,deflate "," Accept-language ":" zh-cn,zh;q=0.8 "}; #int main () #{req = Request. Request (Url=html_url, headers=header); resp = Request.urlopen (req); # return 0;#}

Message header to send after spoofing

get/u012870721 http/1.1 Host:blog.csdn.net connection:close upgrade-insecure-requests:1 user-agent:mozilla/5.0 (Wind OWS NT 10.0; Win64; x64) applewebkit/537.36 (khtml, like Gecko) chrome/60.0.3112.113 safari/537.36 accept:text/html,application/xhtml+xml , application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 accept-encoding:gzip,deflate accept-language:zh-cn,zh;q= 0.8