Python is based on nginx access logs and counts IP traffic

If you want to see how many IP accesses Nginx has, which countries visit, and show the attribution of IP address distribution, Python can use the GAO Chunhui teacher ipip.net "Free IP Address Database", the shell can use Nali, I mainly use the Python language to implement the requirements, and send the query results as mail, but also to learn and review the Python language. Thank you for the free version of the IP address database provided by teacher Gao Chunhui.

One, the Ningx log is as follows:

41.42.97.104 - - [26/feb/2015:03:35:40 -0500]  "get /root/ http/1.1"   301 20  "http://baibai.123.com/09"   "mozilla/5.0  (WINDOWS NT 6.3; WOW64)  AppleWebKit/537.36  (Khtml, like gecko)  chrome/40.0.2214.115 safari/537.36 "  - 0.562 41.42.97.104 - - [26/Feb/2015:03:35:41 -0500]  "get / crossadkla.xml http/1.1 " 304 0 " https://baibai.123.com/" " mozilla/5.0  (Windows  NT 6.3; WOW64)  AppleWebKit/537.36  (Khtml, like gecko)  chrome/ 40.0.2214.115 safari/537.36 " - 0.000 99.122.189.203 - - [26/feb/ 2015:03:35:42 -0500]  "get /root/ http/1.1"  301 20  "HTTP://BAIBAI.123.COM/11 " " mozilla/5.0  (WINDOWS NT 6.3; WOW64)  AppleWebKit/537.36  (khtml, like  gecko)  chrome/40.0.2214.115 safari/537.36 " - 0.562 99.122.189.203  - - [26/feb/2015:03:35:44 -0500]   "get /crossadkla.xml http/1.1"  304 0  "https://baibai.123.com/"   "mozilla/ 5.0  (WINDOWS NT 6.3; WOW64)  AppleWebKit/537.36  (Khtml, like gecko)  chrome/40.0.2214.115 safari/537.36 " - 0.00099.122.189.203  - - [26/ feb/2015:03:35:44 -0500]  "get /crossadkla.xml http/1.1"  304 0  "https:// baibai.123.com/" " mozilla/5.0  (WINDOWS NT 6.3; WOW64)  AppleWebKit/537.36  ( Khtml, like gecko)  chrome/40.0.2214.115 safari/537.36 " - 0.000

Second, download the free version of the IP address database

#wget Http://s.qdcdn.com/17mon/17monipdb.zip #unzip 17monipdb.zip

Three, IP library frequently asked Questions FAQ

Example code:

Import osfrom ipip import ipfrom ipip import ipxip.load (Os.path.abspath ("Mydata4vipday2.dat")) Print Ip.find (" 118.28.8.8 ") Ipx.load (Os.path.abspath (" Mydata4vipday2.datx ")) Print Ipx.find (" 118.28.8.8 ")

Execution output:

China Tianjin Tianjin Peng Dr. China Tianjin Tianjin Peng Dr. 39.128399 117.185112 Asia/shanghai utc+8 120000

IP Library Guihub Address: Https://github.com/17mon/python

Four, Python statistics code

#encoding =utf8import re,sys,os,csv,smtplibfrom ipip import ipfrom ipip import  IPXfrom email import encodersfrom email.mime.multipart import  mimemultipartfrom email.mime.base import mimebasefrom email.mime.text import  Mimetextfrom optparse import optionparserreload (SYS) sys.setdefaultencoding (' Utf-8 ') print  Sys.getdefaultencoding () nginx_log_path= "/app/nginx/logs/apptest_www.access.log" Pattern = re.compile ( R ' ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} ') Def stat_ip_views (Log_path):    ret={}     f = open (log_path,  "R")     for line in f:         match = pattern.match (line)          if match:             Ip=match.group (0)             if ip in ret:                 views=ret[ip]             else:                 views=0             views=views+1            ret[ip]=views     return ret    def run ():     ip_views =stat_ip_views (Nginx_log_path)     max_ip_view={}    filename= ' Out.csv '     f=open (' out.csv ', ' w+ ')     b =  ' IP, country, total number of visits '      print >> f,b    for ip in ip_views:         ip.load (Os.path.abspath ("17monipdb.dat"))          Count=ip.find ("%s"%  (IP))         conut_s=count.split ()          countery=conut_s[0]         views=ip_views[ip]        c =  '%s,%s,%s '  % (Ip,countery, views)         print >> f,c         if len (Max_ip_view) ==0:             max_ip_view[ip]=views        else:             _ip=max_ip_view.keys () [0]             _views=max_ip_view[_ip]             if views>_views:                 max_ip_view[ip]=views                 max_ip_view.pop (_IP)         print  "IP:",  ip,  "Country:", countery,  "Visits:", views              print  "Total number of IPs:",  len (ip_views)     print  " Maximum IP access: ", max_ip_view    g = " "    d = " Total number of ip:%s '  % (len (ip_views))     e =  ' Maximum IP access:%s '  % (max_ip_view)      print >> f,g    print >> f,d     print >> f,edef sendmail (html,emailaddress,mailsubject,from_address= "[Email  protected] "):        mail_list=emailaddress.split (",")          msg=mimemultipart ()         msg[' Accept-Language ']= ' Zh-CN '         msg[' accept-charset ']=  ' iso-8859-1,utf-8 '          msg[' from ']=from_address         msg[' to ']= ";". Join (mail_list)         msg[' Subject ']=mailsubject.decode ("Utf-8")         txt=mimetext (HTML, ' HTML ', ' Utf-8 ')          txt.set_charset (' Utf-8 ')         msg.attach (txt )         file=mimebase (' Application ',  ' Octet-stream ')          file.set_payload (Open (filename,  ' RB '). Read ())          encoders.encode_base64 (file)         file.add_header (' Content-disposition ',  ' attachment; filename= "%s" '  % os.path.basename (filename)          msg.attach (file)         smtp =smtplib. SMTP ("mail.test.com")         smtp.sendmail (msg["from"],mail_list, Msg.as_string ())         smtp.close () if __name__ ==  ' __ Main__ ':     run ()     filename= ' Out.csv '     cmd  =  ' ICONV&NBSP;-F&NBSP;UTF8&NBSP;-T&NBSP;GB18030&NBSP;%S&NBSP;-O&NBSP;%S.BAK&NBSP;&AMP;&AMP;&NBSP;MV  %s.bak %s '  % (filename,filename,filename,filename)     os.system (cmd)      Content=  ' dear all: <br> &nbsp;&nbsp;  Annex National IP Access data analysis statistics , please check!   <br> &nbsp;&nbsp;  If you have any questions, please contact me in time! '     Subject =  ' [analysis] National creation data IP Analytics statistics '     sendmail (html=content, Emailaddress= ' [email protected] ', mailsubject=subject)

V. Results of implementation

UTF-8IP:41.42.97.104 Country: Egypt visits: 2ip:99.122.189.203 Country: United States visits: 3 Total number of Ip:2 maximum IP access: {' 99.122.189.203 ': 3}

Vi. Mail Delivery results

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/59/E6/wKioL1Tu5XmwRhZ6AACUl5-Zu80979.jpg "title=" 11.jpg "alt=" Wkiol1tu5xmwrhz6aacul5-zu80979.jpg "/>

This article is from the "& Think Far Dawn" blog, make sure to keep this source http://kling.blog.51cto.com/3320545/1615505

Python is based on nginx access logs and counts IP traffic