Nginx Log Analysis shell script in Linux

When using a virtual host before, look at the Web site run log, found a lot of abnormal malicious access. At that time because they do not have the system permissions can not be able to screen these actions. Now have their own cloud host, front-end time to view logs, and found a lot of malicious access. It is possible to do a simple log analysis tool to mask some of these operations through a recent study of the shell.
First of all, the so-called analytical tools, is certainly based on artificial analysis. Let's take a look at my domain run log:

The code is as follows

Copy Code

78.56.78.115--[21/may/2014:16:54:27 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:30 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:32 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 42.159.83.42--[21/may/2014:16:54:36 +0800] "head/521php.rar http/1.1" 404 0 "-" "-"- 78.56.78.115--[21/may/2014:16:54:36 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:38 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:41 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:45 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:47 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:50 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:53 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:56 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:54:58 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:00 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 42.159.83.42--[21/may/2014:16:55:05 +0800] "head/521php.zip http/1.1" 404 0 "-" "-"- 78.56.78.115--[21/may/2014:16:55:05 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:07 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:11 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:14 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:17 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:21 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:23 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:25 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:27 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:31 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 42.159.83.42--[21/may/2014:16:55:31 +0800] "head/wwwroot.rar http/1.1" 404 0 "-" "-"- 78.56.78.115--[21/may/2014:16:55:33 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:37 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:39 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:41 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:44 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:50 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:52 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 42.159.83.42--[21/may/2014:16:55:56 +0800] "head/wwwroot.zip http/1.1" 404 0 "-" "-"- 78.56.78.115--[21/may/2014:16:55:57 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:55:59 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:56:01 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:56:03 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"- 78.56.78.115--[21/may/2014:16:56:05 +0800] "post/wp-login.php http/1.0" 200 3198 "-" "-"-

It is obvious that there are IP in malicious brute force to crack my login information, and there is an IP in the attempt to download my website program. This kind of IP we compare the normal access log will find that he has no header information, that is, your browser and so on, and general access is either get or post access, and there is head access. So we can add such IP to the firewall or Nginx IP blacklist, of course, there can be other rules, compared to the frequency of visits, 404 of the frequency, of course, some visits you will find that he visited your home page, but did not load your js,css, of course, here also has the effect of caching, However, even the cache will send a 304 acknowledgment. There are, of course, many other malicious access behaviors.
Then, for these rules, we write the processing logic and operation can, write this, we first do a Nginx IP blacklist. I don't have a Linux firewall here anymore, because the firewall operation is not convenient, frequent restart has certain influence, and the Nginx blacklist, is can smooth restart, and is lets specify the IP to display 403 error, but can visit, this gives the other people a chance of the rehabilitation Ah yes, If someone else has access, you can remove this IP blacklist.
Configure Nginx IP blacklist:
In your nginx.conf configuration file, include a blocksip.conf;
blocksip.conf store blacklist IP;
Write like this

The code is as follows	Copy Code
Deny 1.1.1.2; Deny 1.1.1.1; Deny 1.1.1.4; Deny 1.1.3.1; Deny Http://www.111cn.net;

Then we write a shell script that automatically adds, removes IP, and smoothes the Nginx, and sends email notifications

editblocksip.sh

The code is as follows

Copy Code

#!/bin/sh File= "/etc/nginx/conf/blocksip.conf" File2= "/etc/nginx/conf/blocksip.bak" V1=$1 V2=$2 if [$v 1 = "Add"] Then Deny_info= ' Cat $file | grep $ ' If [-Z "$deny _info"] Then ' Echo ' Deny $v 2; ">> $file" Fi else if [$v 1 = "Del"] Then ' Cat $file | Grep-v $ > $file 2 ' ' Cat $file 2 > $file ' Fi Fi '/usr/sbin/nginx-s Reload ' ' Cat $file |mail-s ' edit blocks list ' zhangcunchao@izptec.com zhangcunchao_cn@163.com ' Exit 0

Easier to use

The code is as follows	Copy Code
#添加 SH editblocksip.sh Add 1.1.1.1 #移除 SH editblocksip.sh del 1.1.1.1

Finally, we just need to write a script to analyze the Nginx log file, have a qualifying record, to trigger this shell script to add IP blacklist
The idea is this, first log the maximum line number, then, each time the parse script is executed, the remaining records are read from the specified line number (tail-n +5 file) and the maximum line number (WC) is recorded again, and then awk is used to parse the record line by row, and the characters are matched to match the criteria, and the IP is extracted, Add IP blacklist;
This analysis script can be run with crontab or daemon. According to the operation of their own site, define the frequency of execution, a few seconds, a few minutes and so on.
This will achieve a simple log analysis tool!