Python Regular parsing Nginx log

There is a need to analyze nginx log, also lazy to study logstach and other open source tools, simply write a script, their own according to the needs to achieve:

Look at the log format first: We are not the same as others, so there is no way:

12.195.166.35 [10/may/2015:14:38:09 +0800] "list.xxxx.com" "Get/new/10:00/9.html?cat=0,0&sort=price_asc http/ 1.0 "42164" Http://list.zhonghuasuan.com/new/10:00/8.html?cat=0,0&sort=price_asc "" mozilla/5.0 (Linux; U Android 4.4.2; ZH-CN; H60-l02 build/hdh60-l02) applewebkit/534.30 (khtml, like Gecko) version/4.0 ucbrowser/10.4.0.558 U3/0.8.0 Mobile Safari/ 534.30 "

Above is my log format:

The script is as follows:

#!/usr/bin/env python#-*- coding:utf-8 –*-#Author: Xiaoluo#qq:942729042#date:2015:05:12import  reimport syslog = sys.argv[1]ip = r "? P<ip>[\d.] * "Date = r"? P<date>\d+ "Month = r"? p<month>\w+ "Year = r"? P<year>\d+ "Log_time = r"? p<time>\s+ "Timezone = r" ""? p<timezone>                  [^\ "]*         " "" Name = r "" "? P<name>\ "            [^\"]*\ "               "" Method = r "? p<method>\s+ "Request = r"? p<request>\s+ "Protocol = r"? p<protocol>\s+ "Status = r"? P<status>\d+ "Bodybytessent = r"? p<bodybytessent>\d+ "Refer = r" "" "? P<refer>\ "             [^\"]*\ "               "" "Useragent=r" ""? p<useragent>                 .*                "" "#f  =  open (' Access1.log ', ' R ') #for  logline in f.readlines ():p  = re.compile (r "(%s) \  \[(%s)/(%s)/(%s) \:(%s) \  (%s) \  (%s) \  (%s) \  (%s) \  (%s) \  (%s) \  (%s) \   (%s) \  (%s) " % (ip, date, month, year, log_time,timezone,name,method,request , protocol,status,bodybytessent,refer,useragent),  re. VERBOSE) Def getcode ():     codedic={}    f = open (log, ' r ')     for logline in f.readlines ():         matchs = p.match (logline)           if matchs !=None:              allgroups =matchs.groups ()               status= allGroups[10]              codedic[status]=codedic.get (status,0)  +1    return codedic     f.close () Def getip ():     f = open (log, ' R ')       ipdic={}    for logline in f.readlines ():         matchs = p.match (logline)          if matchs !=None:             Allgroups =matchs.groups ()             ip=allgroups[0]              ipdic[ip] = ipdic.get (IP,0)  +1     ipdic=sorted (Ipdic.iteritems (), key=lambda c:c[1],reverse=true)     ipdic= Ipdic[0:21:1]    return ipdic    f.close () Def getURL ():     f = open (log, ' R ')     URLdic={}     For logline in f.readlines ():         matchs =  p.match (logline)         if matchs !=None:             allgroups =matchs.groups ()              urlname = allGroups[6]              urldic[urlname] = urldic.get (urlname,0)  +1     Urldic=sorted (Urldic.iteritems (), key=lambda c:c[1],reverse=true)     urldic=urldic[ 0:21:1]               return  URLDICDEF&NBSP;GETPV ():     f = open (log, ' R ')     pvdic={}     for logline in f.readlines ():         matchs = p.match (logline)         if matchs ! =none:           allgroups =matchs.groups ()             timezone=allGroups[4]            time = timezone.split (': ')             minute = time[0]+ ":" +time[1]            Pvdic[minute]=pvdic.get (minute,0)  +1    pvdic=sorted (Pvdic.iteritems (), Key=lambda  c:c[1],reverse=true)     pvdic=pvdic[0:21:1]    return  pvdicif __name__== ' __main__ ':    print  "website Monitoring status Code"      Print getcode ()      print  "website 20 most visited IP address"     print  getip ()     print  "site access up to 20 site names"     print geturl () &NBSP;&NBSP;&NBSP;&NBSP;PRINT&NBSP;GETPV ()

The point here is that. I was to give the regular match when the individual package a function, so that the following each function to open a separate file before opening, but I return can only be returned in the form of a list, the result list is too much my memory consumption, I am 32G of memory, 15G of the log.

Effect:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/6C/D0/wKioL1VTFI2jFdrgAAMgZdOMGos474.jpg "title=" Dasda.png "alt=" Wkiol1vtfi2jfdrgaamgzdomgos474.jpg "/>

The last function is to count the number of accesses per minute

This article is from the "Little Luo" blog, please be sure to keep this source http://xiaoluoge.blog.51cto.com/9141967/1651021

Python Regular parsing Nginx log