Python基於nginx訪問日誌並統計IP訪問量

標籤：

如果想看看Nginx有多少IP訪問量，有哪些國家訪問，並顯示IP地址的歸屬地分布，python可以結合使用高春輝老師ipip.net 【 免費版 IP 位址資料庫 】，Shell可以使用nali，我這邊主要使用python語言來實現需求,並將查詢結果以郵件形式發送，也是為了學習和回顧python語言。很感謝高春輝老師提供的免費版IP地址資料庫。

一、Ningx日誌如下：

41.42.97.104 - - [26/Feb/2015:03:35:40 -0500] "GET /root/ HTTP/1.1" 301 20 "http://baibai.123.com/09" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36" - 0.562 41.42.97.104 - - [26/Feb/2015:03:35:41 -0500] "GET /crossadkla.xml HTTP/1.1" 304 0 "https://baibai.123.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36" - 0.000 99.122.189.203 - - [26/Feb/2015:03:35:42 -0500] "GET /root/ HTTP/1.1" 301 20 "http://baibai.123.com/11" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36" - 0.562 99.122.189.203  - - [26/Feb/2015:03:35:44 -0500] "GET /crossadkla.xml HTTP/1.1" 304 0 "https://baibai.123.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36" - 0.00099.122.189.203  - - [26/Feb/2015:03:35:44 -0500] "GET /crossadkla.xml HTTP/1.1" 304 0 "https://baibai.123.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36" - 0.000

二、下載 免費版 IP 位址資料庫

 #wget  http://s.qdcdn.com/17mon/17monipdb.zip #unzip  17monipdb.zip

三、IP庫常見問題FAQ

範例程式碼：

import osfrom ipip import IPfrom ipip import IPXIP.load(os.path.abspath("mydata4vipday2.dat"))print IP.find("118.28.8.8")IPX.load(os.path.abspath("mydata4vipday2.datx"))print IPX.find("118.28.8.8")

執行輸出：

中國  天津  天津      鵬博士中國  天津  天津      鵬博士   39.128399   117.185112  Asia/Shanghai   UTC+8   120000

IP庫guihub地址： https://github.com/17mon/python

四、Python 統計代碼

#encoding=utf8import re,sys,os,csv,smtplibfrom ipip import IPfrom ipip import IPXfrom email import encodersfrom email.mime.multipart import MIMEMultipartfrom email.mime.base import MIMEBasefrom email.mime.text import MIMETextfrom optparse import OptionParserreload(sys)sys.setdefaultencoding(‘utf-8‘)print sys.getdefaultencoding()nginx_log_path="/app/nginx/logs/apptest_www.access.log"pattern = re.compile(r‘^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}‘)def stat_ip_views(log_path):  ret={}  f = open(log_path, "r")  for line in f:    match = pattern.match(line)    if match:      ip=match.group(0)      if ip in ret:        views=ret[ip]      else:        views=0      views=views+1      ret[ip]=views  return retdef run():  ip_views=stat_ip_views(nginx_log_path)  max_ip_view={}  fileName=‘out.csv‘  f=open(‘out.csv‘,‘w+‘)  b = ‘IP,國家,訪問數總數‘  print >> f,b  for ip in ip_views:    IP.load(os.path.abspath("17monipdb.dat"))    count=IP.find("%s"% (ip))    conut_s=count.split()    countery=conut_s[0]    views=ip_views[ip]    c = ‘%s,%s,%s‘ %(ip,countery,views)    print >> f,c    if len(max_ip_view)==0:      max_ip_view[ip]=views    else:      _ip=max_ip_view.keys()[0]      _views=max_ip_view[_ip]      if views>_views:        max_ip_view[ip]=views        max_ip_view.pop(_ip)    print "IP:", ip, "國家:", countery, "訪問數:", views   print "總共有多少IP:", len(ip_views)  print "最大訪問IP數:", max_ip_view  g = ""  d = ‘總共有多少IP:%s‘ %(len(ip_views))  e = ‘最大訪問IP數:%s‘ %(max_ip_view)  print >> f,g  print >> f,d  print >> f,edef sendMail(html,emailaddress,mailSubject,from_address="[email protected]"):    mail_list=emailaddress.split(",")    msg=MIMEMultipart()    msg[‘Accept-Language‘]=‘zh-CN‘    msg[‘Accept-Charset‘]= ‘ISO-8859-1,utf-8‘    msg[‘From‘]=from_address    msg[‘to‘]=";".join(mail_list)    msg[‘Subject‘]=mailSubject.decode("utf-8")    txt=MIMEText(html,‘html‘,‘utf-8‘)    txt.set_charset(‘utf-8‘)    msg.attach(txt)    file=MIMEBase(‘application‘, ‘octet-stream‘)    file.set_payload(open(fileName, ‘rb‘).read())    encoders.encode_base64(file)    file.add_header(‘Content-Disposition‘, ‘attachment; filename="%s"‘ % os.path.basename(fileName))    msg.attach(file)    smtp=smtplib.SMTP("mail.test.com")    smtp.sendmail(msg["From"],mail_list,msg.as_string())    smtp.close()if __name__ == ‘__main__‘:  run()  fileName=‘out.csv‘  cmd = ‘iconv -f UTF8 -t GB18030 %s -o %s.bak && mv %s.bak %s‘ %(fileName,fileName,fileName,fileName)  os.system(cmd)  Content= ‘Dear ALL: <br>附件內國家IP訪問資料分析統計，請查收！  <br>如有任何問題，請及時與我聯絡！‘  Subject = ‘[分析]國家建立資料IP分析統計‘  sendMail(html=Content,emailaddress=‘[email protected]‘,mailSubject=Subject)

五、執行結果

utf-8IP: 41.42.97.104 國家: 埃及 訪問數: 2IP: 99.122.189.203 國家: 美國 訪問數: 3總共有多少IP: 2最大訪問IP數: {‘99.122.189.203‘: 3}

六、郵件發送結果

• 本文來自：Linux教程網

Python基於nginx訪問日誌並統計IP訪問量