Python combat: How to hide your reptile identity

Using crawlers to access the site, you need to hide their identities as much as possible, in case of being blocked by the server, in the work of the project, we have 2 ways to achieve this goal, are delayed access and dynamic agents, and then we will explain these two ways

1. Delayed access

In the sense of name, time-lapse access is to set an access cycle when visiting a website, accessed every few seconds, in a way that is more like an artificial visit to the site

   
  
 
   

   
Import Time
 
   

   
Import Urllib.request
 
   

   
 
   

   
CNT = 0
 
   

   
#隐藏自己爬虫的身份的第一种策略是设置访问周期, making the program more like an artificial access
 
   

   
 while True: #每隔5秒钟访问一次百度网
 
   

   
url = "https://www.baidu.com" #设置url地址
 
   

   
param = {} #设置参数, parameter is dictionary
 
   

   
param = Urllib.parse.urlencode (param). Encode (' utf_8 ') #将参数以utf-encoded by 8 encoding
 
   

   
 
 
   

   
req = urllib.request.Request (URL, param)
 
   

   
 #设置header的User the-agent property, impersonate the request is sent by the Fox Fire Browser, that is, the spoofing server is sent by the sender is not sent by the program
 
   

   
Req.add_header ("User-agent", "mozilla/5.0" (Macintosh; Intel Mac OS X 10.12; rv:53.0) gecko/20100101 firefox/53.0 ")
 
   

   
Response = Urllib.request.urlopen (req) #访问网络
 
   

   
 
 
   

   
html = response.read () #读取响应的结果
 
   

   
result = Html.decode ("Utf-8") #按照utf-8 encoding for decoding
 
   

   
 if result! = "":
 
   

   
CNT + = 1
 
   

   
Print ("%s attacks Baidu Net" %cnt)
 
   

   
Time.sleep (5) #程序睡眠5秒钟
 
  

Operation Result:

Visit Baidu net every 5 seconds

2. Dynamic Agent

Use proxy server to access the site, this method is very overbearing, can simulate different server access to the site, but also the most recommended way, we can find a free proxy server IP on Baidu online

   
  
 
   

   
Import Urllib.request
 
   

   
Import Random
 
   

   
 
   

   
IPList = [' 119.6.144.73:81 ', ' 183.203.208.166:8118 ', ' 111.1.32.28:81 '] #定义多个代理IP, proxy ip can be searched online for free
 
   

   
CNT = 0
 
   

   
#隐藏自己爬虫的身份的第二种策略是使用代理, which means to simulate multiple server accesses
 
   

   
 while True: #使用代理服务器不停的访问百度网
 
   

   
Proxy_support = Urllib.request.ProxyHandler ({' http ': Random.choice (IPList)}) #定义一个代理对象, using a random IP 
 
   

   
 
 
   

   
Opener = Urllib.request.build_opener (Proxy_support)
 
   

   
Opener.add_handlers = [("User-agent", "mozilla/5.0" (Macintosh; Intel Mac OS X 10.12; rv:53.0) gecko/20100101 firefox/53.0 ")]
 
   

   
Urllib.request.install_opener (opener)
 
   

   
 
 
   

   
Response = Urllib.request.urlopen ("https://www.baidu.com") #访问网络
 
   

   
 
 
   

   
html = response.read () #读取响应的结果
 
   

   
result = Html.decode ("Utf-8") #按照utf-8 encoding for decoding
 
   

   
 if result! = "":
 
   

   
CNT + = 1
 
   

   
Print ("%s attacks Baidu Net" %cnt)
 
  

Operation Result:

Keep attacking the Baidu network

Python combat: How to hide your reptile identity