9-python Proxyhandler Processor (proxy settings)

Proxyhandler processor (proxy settings)

Using proxy IP, this is the second trick of the crawler/anti-crawler, and is usually the best.

Many websites will detect the number of times an IP is visited (through traffic statistics, system logs, etc.), and if the number of visits is not as normal, it will prohibit access to this IP.

So we can set up some proxy server, change an agent every time, even if the IP is forbidden, can still change the IP to continue crawling.

#_ *_ coding:utf-8 _*_ "Created on July 13, 2018 @author:sss function: Use proxy" "Import urllib.requestimport randomproxy_list = [{" HTTP ":" 124.88.67.81:80 "}, {" http ":" 124.88.67.81:80 "}, {" http ":" 124.88.67.81:80 "}, {" http ":" 124.88.67.81:8 0 "}, {" http ":" 124.88.67.81:80 "}]proxy = Random.choice (proxy_list) #构建两个代理Handler, one with proxy IP, one without proxy ip# httpproxy_ Handler = Urllib.request.ProxyHandler ({' http ': ' 124.193.85.88:8080 '}) Httpproxy_handler = Urllib.request.ProxyHandler (proxy) #每次随机一个代理httpProxy_handler = Urllib.request.ProxyHandler ({}) # Even if five IP has an empty dictionary Proxyswitch = True #定义一个代理开关 # proxyswitch = False #定义一个代理开关 # Use Proxy handler object to create custom opener object # Depending on whether the proxy switch is turned on, Use a different proxy mode if Proxyswitch:opener = Urllib.request.build_opener (httpproxy_handler) Else:opener = Urllib.request.build_ Opener (httpproxy_handler) request = Urllib.request.Request (' http://www.baidu.com/') # 1. If you write this, only use Opener.open () A custom proxy can be used when sending a request, while using Urlopen () is not using a custom proxy response = Opener.open (Request) # 2. If this is the case, the opener is applied to the global, then all, whether opener. open () or urlopen () send requests are all using custom proxies # Urllib.request.install_opener (opener) # respense = Urllib.request.urlopen (Request) Print (Response.read ()) print (' finish! ')

　　

• West Thorn Free Proxy IP
• Quick Agent Free agent
• PROXY360 Agent
• Full Network proxy IP

9-python Proxyhandler Processor (proxy settings)