When using Python crawlers, we sometimes use IP proxies. Inadvertently found a free proxy IP website: http://www.xicidaili.com/nn/. However, found that a lot of IP is not used. So a script is written in Python that can detect the proxy IP that can be used. The script is as follows:
#encoding =utf8 import urllib2 from BS4 import beautifulsoup import urllib import Socket user_agent = ' mozilla/5.0 (Window S NT 6.3; WOW64;
rv:43.0) gecko/20100101 firefox/43.0 ' header = {} header[' user-agent '] = User_agent ' Get all proxy IP address ' def getproxyip (): Proxy = [] for I in range (1,2): Try:url = ' http://www.xicidaili.com/nn/' +str (i) r EQ = urllib2.
Request (url,headers=header) res = Urllib2.urlopen (req). Read () soup = beautifulsoup (res) ips = Soup.findall (' tr ') for X in range (1,len (IPS)): IP = ips[x] TDs = ip.fi
Ndall ("td") Ip_temp = tds[1].contents[0]+ "\ t" +tds[2].contents[0] proxy.append (ip_temp) Except:continue return proxy ' verifies that the obtained proxy IP address is available ' Def validateip (proxy): url = "Http://ip.c Hinaz.com/getip.aspx "f = open (" E:\ip.txt "," W ") Socket.setdefaulttimeout (3) for I in range (0,len (proxy)): Try:ip = Proxy[i].strip (). Split ("T") Proxy_host = "http://" +ip[0]+ ":" +ip[1] proxy_temp = {"http":p roxy_host} res = Urllib.urlopen (url,proxies=proxy_temp). Read () F.write (pro xy[i]+ ' \ n ') print proxy[i] except exception,e:continue f.close () if __name__ = = ' __main__ ': Proxy = Getproxyip () validateip (proxy)
After running successfully, open the file under E disk, you can see the following available proxy IP address and port:
This is just crawling the first page of the IP address, if necessary, you can crawl more than a few pages. At the same time, the site is always updated, it is recommended to crawl only the first few pages can be crawled.