Python code example for creating a web traffic tool

This article will share with you a small tool made in python that can be used to fl web page traffic and provide you with detailed code, if you have any need, you can refer to this article to share with you a small tool made in python that allows you to brush web page traffic and provide detailed code, if you have any friends, refer

Preparation

Required environment:

Python3

Start

First, implement a simple version and directly add the code:

Import urllib. requestimport urllib. error # create get method def get (url): code = urllib. request. urlopen (url ). code return codeif _ name _ = '_ main _': # set some basic attributes url = "http://shua.jb51.net" user_agent = "Mozilla/5.0 (Windows NT 10.0; win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 "headers = {'user-Agent': user_agent} req = urllib. request. request (url, headers = headers) # Record times I = 1 while 1: code = get (url) print ('Access: '+ str (code) I = I + 1

Simple and crude, only pv is flushed, and the ip address is not changed, which is easy to be found by search engines. let's improve it.

Added proxy function

Add the following code to the get method:

random_proxy = random.choice(proxies)proxy_support = urllib.request.ProxyHandler({"http":random_proxy})opener = urllib.request.build_opener(proxy_support)urllib.request.install_opener(opener)

Modify the main method:

If _ name _ = '_ main _': url = "http://shua.jb51.net" # Add proxy list, you can go to Baidu to obtain proxies = ["124.88.67.22: 80 ", "124.88.67.82: 80", "124.88.67.81: 80", "124.88.67.31: 80", "124.88.67.19: 80", "58.23.16.240: 80 "] user_agent =" Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) chrome/51.0.2704.63 Safari/537.36 "headers = {'user-Agent': user_agent} req = urllib. request. request (url, headers = headers) I = 1 while 1: # add the parameter code = get (url, proxies) print ('dide' + str (I) + 'proxy access: '+ str (code) I = I + 1

This is almost the case, but there is a bug. if the page cannot be opened or the proxy fails, the program will automatically end. next we will add the exception handling function.

Exception handling

Define the mail method to send an email notification.

Def mail (txt): _ user = "your account" _ pwd = "your password" _ to = "recipient account" msg = MIMEText (txt, 'plain ', 'utf-8') # The title msg ["Subject"] = "proxy is invalid! "Msg [" From "] = _ user msg [" To "] = _ to try: # here, my QQ mailbox s = smtplib. SMTP_SSL ("smtp.qq.com", 465) s. login (_ user, _ pwd) s. sendmail (_ user, _ to, msg. as_string () s. quit () print ("Success! ") Cipher T smtplib. SMTPException as e: print (" Falied, % s "% e)

Then let's modify the main method:

If _ name _ = '_ main _': url = "http://shua.jb51.net" proxies = ["124.88.67.22: 80", "124.88.67.82: 80", "124.88.67.81: 80 "," 124.88.67.31: 80 "," 124.88.67.19: 80 "," 58.23.16.240: 80 "] user_agent =" Mozilla/5.0 (Windows NT 10.0; Win64; x64) appleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 "headers = {'user-Agent': user_agent} req = urllib. request. request (url, headers = headers) I = 1 while 1: try: code = get (url, proxies) print ('dide' + str (I) + 'proxy access: '+ str (code) I = I + 1 records T urllib. error. HTTPError as e: print (e. code) # add the mail method mail (e. code) failed t urllib. error. URLError as err: print (err. reason) # add the mail method mail (err. reason)

Done!

Conclusion

There are only 50 lines of code, and the program can be improved:

For example, the agent list is automatically obtained, the interface is added, and multithreading is extended.

Finally, I would like to share my work with other friends.

import urllib2import timeitimport thread import timei = 0mylock = thread.allocate_lock()def test(no,r):  global i  url = 'http://blog.csdn.net'  for j in range(1,r):    req=urllib2.Request(url)     req.add_header("User-Agent","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)")     file = urllib2.urlopen(req)    print file.getcode();    mylock.acquire()    i+=1    mylock.release()      print i;  thread.exit_thread()def fast():    thread.start_new_thread(test,(1,50))    thread.start_new_thread(test,(2,50)) fast()time.sleep(15)

After testing, a 503 error occurs on servers with more than two threads, so the two threads are

The above is the detailed content of the code instance for the Python web traffic tool. For more information, see other related articles in the first PHP community!