linux中nginx日誌分析shell指令碼

以前使用虛擬機器主機的時候，查看網站作業記錄，就發現了很多異常的惡意的訪問。當時因為自己並沒有系統許可權沒辦法對這些行為進行屏蔽。現在有了自己的雲主機，前端時間查看日誌，又發現了很多惡意訪問。正好可以通過最近對shell的學習，來做一個簡單的日誌分析工具，來屏蔽一些這樣的操作。
首先，所謂的分析工具，肯定是建立在人為的分析的基礎上的。我們來看一點我的網域名稱作業記錄：

代碼如下

複製代碼

78.56.78.115 - - [21/May/2014:16:54:27 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:30 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:32 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 42.159.83.42 - - [21/May/2014:16:54:36 +0800] "HEAD /521php.rar HTTP/1.1" 404 0 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:36 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:38 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:41 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:45 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:47 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:50 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:53 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:56 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:54:58 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:00 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 42.159.83.42 - - [21/May/2014:16:55:05 +0800] "HEAD /521php.zip HTTP/1.1" 404 0 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:05 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:07 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:11 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:14 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:17 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:21 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:23 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:25 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:27 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:31 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 42.159.83.42 - - [21/May/2014:16:55:31 +0800] "HEAD /wwwroot.rar HTTP/1.1" 404 0 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:33 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:37 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:39 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:41 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:44 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:50 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:52 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 42.159.83.42 - - [21/May/2014:16:55:56 +0800] "HEAD /wwwroot.zip HTTP/1.1" 404 0 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:57 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:55:59 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:56:01 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:56:03 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" - 78.56.78.115 - - [21/May/2014:16:56:05 +0800] "POST /wp-login.php HTTP/1.0" 200 3198 "-" "-" -

很明顯可以看到有ip在惡意暴力破解我的登入資訊，還有一個ip在嘗試下載我的網站程式。這樣的ip我們對比正常的訪問日誌會發現，他沒有頭資訊，即你瀏覽器等等的資訊，而且一般訪問都是get或者post訪問，而這裡有head訪問。所以我們可以將這樣的ip加入防火牆或者nginx的ip黑名單，當然還可以有其他規則，比較訪問的頻次，404出現的頻次，當然有的訪問你會發現他訪問你的首頁，卻沒有載入你的js，css等，當然這裡也有緩衝的影響，但是即使緩衝也會發送304的確認資訊。當然還有很多其他的惡意訪問行為。
然後，針對這些規則，我們來寫處理邏輯和操作即可，寫這個之前，我們先做一個nginx的ip黑名單。我這裡沒有再使用linux的防火牆，因為防火牆操作不太方便，頻繁重啟有一定的影響，而nginx的黑名單，是可以平滑重啟的，而且是讓指定ip顯示403錯誤，但是可以訪問的，這樣就給人家一個改過自新的機會啊是吧，如果人家訪問正常了，就可以將此ip移除黑名單。
配置nginx ip黑名單：
在你的nginx.conf設定檔中，包含一個blocksip.conf;
blocksip.conf 存放黑名單ip；
這樣寫

代碼如下	複製代碼
deny 1.1.1.2; deny 1.1.1.1; deny 1.1.1.4; deny 1.1.3.1; deny http://www.111cn.net;

然後我們寫個shell指令碼，自動添加、移除ip，並平滑重啟nginx，並發送郵件通知

editblocksip.sh

代碼如下

複製代碼

#!/bin/sh file="/etc/nginx/conf/blocksip.conf" file2="/etc/nginx/conf/blocksip.bak" v1=$1 v2=$2 if [ $v1 = "add" ] then   deny_info=`cat $file | grep $2`   if [ -z "$deny_info" ]   then   `echo "deny $v2;" >> $file`   fi else if [ $v1 = "del" ] then   `cat $file | grep -v $2 > $file2`   `cat $file2 > $file` fi fi `/usr/sbin/nginx -s reload` `cat $file|mail -s "edit blocks list" zhangcunchao@izptec.com zhangcunchao_cn@163.com` exit 0

使用比較簡單

代碼如下	複製代碼
#添加 sh editblocksip.sh add 1.1.1.1   #移除 sh editblocksip.sh del 1.1.1.1

最後，我們只需要寫個指令碼，來分析nginx記錄檔，有合格記錄，來觸發這個shell指令碼添加ip黑名單即可
思路是這樣的，首先記錄日誌最大行號，然後每次執行分析指令碼，從指定行號讀取剩下所有記錄（tail -n +5 file），並再次記錄最大行號（wc）,然後使用awk，逐行分析記錄，並切割字元進行匹配判斷，合格，提取ip，加入ip黑名單；
這個分析指令碼，可以用crontab 或者 做成守護進程來跑。根據自己網站的運行情況，定義執行頻次，幾秒，幾分鐘等等。
這樣就實現了一個簡單的日誌分析工具！