This article mainly introduces how to use the proxy server when based on scrapy. it involves the skills of using the proxy server in Python and has some reference value, for more information about how to use the proxy server when from scrapy, see the example in this article. Share it with you for your reference. The details are as follows:
# To authenticate the proxy, # you must set the Proxy-Authorization header. # You * cannot * use the form http: // user: pass @ proxy: port # in request. meta ['proxy'] import base64proxy_ip_port = "123.456.789.10: 8888" proxy_user_pass = "awesome: dude" request = Request (url, callback = self. parse) # Set the location of the proxyrequest. meta ['proxy'] = "http: // % s" % proxy_ip_port # setup basic authentication for the proxyencoded_user_pass = base64.encodestring (proxy_user_pass) request. headers ['proxy-authorization'] = 'basic '+ encoded_user_pass
I hope this article will help you with python programming.