Linux server automatically blocks abnormal access IP scripts (1)

Linux server automatically blocks abnormal access IP scripts (1)
Problem:
Recently, we have found many crawlers crawling some of our website interfaces with high concurrency, leading to frequent bandwidth exceptions and service exceptions. To temporarily get lazy, write scripts to automatically block these IP addresses.
Ideas and methods:
After log troubleshooting, it is found that there are not many crawler IP addresses, but the concurrency is very large, basically between 10-20 concurrency, and there are almost 10 concurrent requests for a normal single IP address, so the solution is
Use iptables to temporarily block these abnormal IP addresses. The blocking rules exceed 3000 requests per IP address every five minutes, that is, if the concurrency of A Single IP address exceeds 10 within five minutes, the IP address is blocked, in addition, to prevent blocking errors, the script automatically bans the IP addresses that were blocked five minutes ago every five minutes.
Implementation:
For simple management, shell + crontab is used for script processing.
Script:

#! /Bin/sh # Author: bbzsxjj # Email: bbzsxjj@163.com # Usage: auto deny ip by iptables # Ver: 1.0 Timenow = 'date + '% H % m00'' Timelast = 'date + '% H % M00'-d'-5min'' Lastnum = 1000000 # Number of log lines, which can be selected based on your business frequency Limitnum = 3000 # concurrency limit, 300*10 Ipbin =/sbin/iptables NeedDenyiplist =/opt/sbin/ipdeny. list NeedPurgeiplist =/opt/sbin/ippurge. list LogFile =/data/logs/haproxy. log CreateList (){ If [-f $ {LogFile}] Then If [-f $ {NeedDenyiplist}] Then Mv $ {NeedDenyiplist }$ {NeedPurgeiplist} Fi Tail-n $ {lastnum }$ {LogFile} | awk '{gsub (//, "", $3 ); if ($3 >=' $ {timelast} '& $3 <=' $ {timenow }') {a [$6] ++} END {for (I in a) {if (a [I]> '$ {limitnum }') {print $6 }}' >$ {NeedDenyiplist} # here you need to process the log according to the log format. Modify the processing method according to the actual situation. If ['wc-l $ {NeedDenyiplist} | awk '{print $1} ''-eq 0] Then Rm-rf $ {NeedDenyiplist} Fi Fi } DenyIP (){ For Dip in 'cat $ {NeedDenyiplist} | grep-E-V' ^ $ | #'' Do $ {Ipbin}-I INPUT-s $ {Dip}-p tcp -- dport 80-j DROP Done } PurgeIP (){ For Dip in 'cat $ {NeedPurgeiplist} | grep-E-V' ^ $ | #'' Do $ {Ipbin}-d input-s $ {Dip}-p tcp -- dport 80-j DROP Done } Main (){ CreateList If [-f $ {NeedPurgeiplist}] Then PurgeIP Fi If [-f $ {NeedDenyiplist}] Then DenyIP Fi } Main

Then add crontab:
### Denyip
*/5 */bin/sh/opt/sbin/denyip. sh>/dev/null 2> & 1;