PHANTOMJS Setting up Agents

PHANTOMJS can be configured with the proxy IP

#coding =utf-8
Import OS
Import re
Import time
Import requests
From Scrapy.selector import Htmlxpathselector
From scrapy.http import Htmlresponse
From selenium import Webdriver
From Selenium.webdriver.common.proxy import Proxytype

Import Sys
Reload (SYS)
Sys.setdefaultencoding ("Utf-8")

Import warnings
Warnings.filterwarnings ("Ignore")

if __name__ = = ' __main__ ':
Path_phantomjs=r ' D:\phantomjs\bin\phantomjs.exe '
Browser=webdriver. PHANTOMJS (PATH_PHANTOMJS)
#真实ip
Browser.get (' http://icanhazip.com/') #获取ip网址
Response = Htmlresponse (url= ", Body=str (Browser.page_source))
HXS = Htmlxpathselector (response)
print ' Your IP is: ', '. Join (Hxs.select ('//text () '). Extract ()). Strip ()
#代理ip
Proxy=webdriver. Proxy ()
Proxy.proxy_type=proxytype.manual
Proxy.http_proxy= ' 220.248.229.45:3128 '
#将代理设置添加到webdriver. In Desiredcapabilities.phantomjs
Proxy.add_to_capabilities (Webdriver. DESIREDCAPABILITIES.PHANTOMJS)
Browser.start_session (Webdriver. DESIREDCAPABILITIES.PHANTOMJS)
Browser.get (' http://icanhazip.com/') #获取ip网址
Response = Htmlresponse (url= ", Body=str (Browser.page_source))
HXS = Htmlxpathselector (response)
print ' Your proxy IP is: ', '. Join (Hxs.select ('//text () '). Extract ()). Strip ()

Tested, actually effective, as follows:

Later, you can consider applying this method to the download to reduce the chance of spiders being blocked.

PHANTOMJS Setting up Agents