Small Python Program

Requirements : Statistics of how many lines per minute, and 23:59 how many lines, 00:00 how many lines, file too large each file 2G, total 5 files

files : access_log.2017-11-06-00_00_00 file contents are as follows:

21.123.42.233--[05/nov/2017:23:59:59 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker HTTP/1.1" time:11637 1388 21.123.42.233--[06/nov/2017:00:00:00 +0800] "post/ippsremote/remoting/remotecfcap1v Erifysignmsginvoker http/1.1 "time:7223 21.123.42.235--[05/nov/2017:23:59:59 +0800]" post/ip Psremote/remoting/remoteips35062invoker http/1.1 "time:185121 135 21.123.42.235--[06/nov                        /2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" time:7998 330 21.123.42.232--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker HTTP/1.1 "time:29833 1388 21.123.42.232--[06/nov/2017:00:00:00 +0800]" Post/ippsremote/remoting/remotecfcap 1VerifySignMsgInvoker http/1.1 "time:20608 21.123.42.232--[05/nov/2017:23:59:59 +0800]" POST/ Ippsremote/remoting/remoteipS35062invoker http/1.1 "time:252234 139 21.123.42.232--[05/nov/2017:23:59:59 +0800]" POS T/ippsremote/remoting/remoteips35062invoker http/1.1 "time:230661 135 21.123.42.232--[0               6/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker HTTP/1.1" time:33536 1388 21.123.42.233--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remoteips35062invoker http/1.1" 200 time:138429 139 21.123.42.232--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remot Ecfcap1verifysignmsginvoker http/1.1 "time:24368 21.123.42.233--[06/nov/2017:00:00:00 +0800]"  Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1 "time:7075 330 21.123.42.232-- [06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" time:23221 330 21.123.42.2--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" time:19719 330 21.123.42.232--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remoteips35062invoker HTTP/1 .1 "time:85528 135 21.123.42.232--[06/nov/2017:00:00:00 +0800]" Post/ippsremote/remoti Ng/remotecfcaqrycert4softcertinvoker http/1.1 "time:22324 1388 21.123.42.232--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1" time:15338 150 21.123.42 .232--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker HTTP/1.1": 20461 1388 21.123.42.232--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvo Ker http/1.1 "time:24474 21.123.42.235--[06/nov/2017:00:00:00 +0800]" post/ippsremote/r Emoting/remotecfcaqrycert4sOftcertinvoker http/1.1 "time:10589 1388 21.123.42.232--[06/nov/2017:00:00:00 +0800]" Post/ippsremo Te/remoting/remotecfcap1verifysignmsginvoker http/1.1 "time:24163 21.123.42.235--[06/nov/2017:                  00:00:00 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1" time:4449 150 21.123.42.236--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remoteips35062invoker http/1.1": 345698 139 21.123.42.236--[06/nov/2017:00:00:00 +0800] "Post/ippsremote/remoting/remotecfcap 1SignMsgInvoker http/1.1 "time:19886 21.123.42.233--[06/nov/2017:04:02:04 +0800]" POST/ Ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1 "time:5096 150 21.123.42.232--[0                     6/nov/2017:04:02:04 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker HTTP/1.1" time:5755 150 21.123.42.235--[06/nov/2017:04:02:04 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" time:7182 330 21.123.42.235--[06/nov/2017:04:02:04 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvok Er http/1.1 "time:4752 21.123.42.233--[06/nov/2017:04:02:04 +0800]" Post/ippsremote/remot Ing/remoteips35062invoker http/1.1 "time:101802 139 21.123.42.232--[06/nov/2017:04:02 : +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" time:7666 330 21 .123.42.236--[06/nov/2017:04:02:04 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker HTTP/1.1" time:11735 1388 21.123.42.232--[06/nov/2017:04:02:04 +0800] "POST/IPPSREMOTE/REMOTING/REMOTECFCA P1verifysignmsginvoker http/1.1 "time:5376 21.123.42.234--[06/nov/2017:04:02:04 +0800]" PO St/ippsremote/remoting/remoteCfcaqrycert4softcertinvoker http/1.1 "time:19922 1388 21.123.42.236--[06/nov/2017:04:02:04 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker http/1.1" time:8779 1388 21.123.42.23                               5--[06/nov/2017:04:02:04 +0800] "Post/ippsremote/remoting/remoteips35062invoker http/1.1" time:99890 135 21.123.42.234--[06/nov/2017:04:02:04 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsgi Nvoker http/1.1 "time:5242 21.123.42.236--[06/nov/2017:04:02:04 +0800]" post/ippsremote/r Emoting/remotecfcaqrycert4softcertinvoker http/1.1 "time:16658 1388 21.123.42.233--[06/nov/2017:0  4:02:04 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1" time:4977 150 21.123.42.234 --[06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" time:5559 321.123. 42.232--[06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" time:5176 321.123.42.234--[06/Nov/ 2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker http/1.1" TIME:21.123.42.234-- [06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker http/1.1" time:21.123.42.234--[06/Nov/ 2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker http/1.1" TIME:21.123.42.234-- [06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1" time:21.123.42.234--[06/Nov/ 2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1" time:21.123.42.235--[ 06/nov/2017:12:12:31 +0800] "Post/ippsremote/remoting/remoteips35062invoker http/1.1" time:136333 135 21.123.42.234--[06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcap1signmsginvoker http/1.1" : 8429 321.123.42.233--[06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker HTTP/1.1" : 21.123.42.236--[06/nov/2017:12:12:31 +0800] "Post/ippsremote/remoting/remoteips35062invoker http/1.1": 147556 135 21.123.42.233--[06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1 "time:21.123.42.235--[06/nov/2017:12:12:32 +0800]" Post/ippsremote/remoting/remoteips35062invoker HTTP /1.1 "time:119417 135 21.123.42.233--[06/nov/2017:12:12:31 +0800]" Post/ippsremote/remoting/remoteips35062invoke R http/1.1 "time:194970 135 21.123.42.235--[06/nov/2017:12:12:32 +0800]" post/ippsremote/remoting/remoteips35062i Nvoker http/1.1 "time:76006 135 21.123.42.236--[06/nov/2017:12:12:32 +0800]" POST/IPPSREMOTE/REMOTING/REMOTECFCA P1signmsginvoker http/1.1 "time:5892 321.123.42.233--[06/nov/2017:12:12:32 +0800]" Post/ippsremote/remoting/remo Tecfcap1signmsginvoker http/1.1 "3 time:490421.123.42.235--[06/nov/2017:12:12:31 +0800] "Post/ippsremote/remoting/remoteips35963invoker http/1.1": 297515 369 21.123.42.236--[06/nov/2017:12:12:31 +0800] "Post/ippsremote/remoting/remoteips35062invoker HTTP/1.1" 200 time:402771 135 21.123.42.233--[06/nov/2017:12:12:32 +0800] "post/ippsremote/remoting/ Remotecfcaqrycert4softcertinvoker http/1.1 "time:21.123.42.234--[06/nov/2017:12:12:32 +0800]" post/ippsremote/ Remoting/remoteips35062invoker http/1.1 "time:135743 135 21.123.42.235--[06/nov/2017:12:12:32 +0800]" post/ippsre Mote/remoting/remotecfcap1signmsginvoker http/1.1 "time:5853 321.123.42.235--[06/nov/2017:12:12:32 +0800]" POST/ Ippsremote/remoting/remotecfcap1signmsginvoker http/1.1 "time:14536 21.123.42.233--[06/nov/2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcap1verifysignmsginvoker http/1.1" time:21.123.42.233--[06/Nov/ 2017:12:12:32 +0800] "Post/ippsremote/remoting/remotecfcaqrycert4softcertinvoker HTTp/1.1 "time:21.123.42.233--[06/nov/2017:12:12:31 +0800]" Post/ippsremote/remoting/remoteips35062invoker HTTP/ 　 1.1 "time:186202 135

The Python code looks like this:

#!/usr/bin/env python#-*-coding:utf-8-*-__author__='40kuai'bin= [0 forIinchRange (1440)]with Open ('/root/access_log.2017-11-06-00_00_00') as file: forLineinchFile:hour= Line.split (':') [1] mins= Line.split (':') [2] haha= Int (hour) *60+int (mins) Li[haha]+=1 forIndex,iinchEnumerate (LI):Try: A, b=divmod (index,60)        Print '%s:%s'%(A, B), IexceptZerodivisionerror as E:Print '00:00', I

and a simple shell script with Shell, the script is as follows:

#!/bin/bashfor i in {00..23};d o to  j in {00..59};d o     echo $i: $j  ' awk ' {print $4} '/root/access_log.2017-11-0 6-00_00_00|awk-f ': ' {print $2$3} ' |grep $i $j|wc-l ' >>/lines.txt  donedone

The efficiency of the two scripts varies greatly ... Which great God can help to see if we can optimize the shell script, the shell main problem is to filter the loop read file

Small Python Program