Anti-DDoS: CC attack defense system deployment

Anti-DDoS: CC attack defense system deployment
1. System effect this DDOS Application Layer defense system has been deployed on the http://www.yfdc.org site (if access fails, please directly access the server in China http: // 121.42.45.55 for online testing ). The defense system is at the application layer, which effectively prevents the abuse of server resources by illegal users:

As long as it sends high-frequency and application-layer requests to achieve a large amount of system resource consumption attacks, it can effectively defend against attacks.

The basic idea of its implementation is:

Periodically analyze the request frequency of all access users in the past period of time. If the request frequency is higher than the specified threshold, the user is identified as a resource abuse user. After the validity period, the defense system automatically unblocks it.

Online performance testing:

Go to http://www.yfdc.org-> click the upper right sideOnline queryIn/shell/yfDomain,/shell/yfYesbash scriptsWrite Dynamicsweb-cgiProgram, the user submits information each time, this program will execute some server-side query operations, and then return the data processing to the client.

To prevent unauthorized users from accessing this program frequently and affect the access of other normal users, some protection measures are required.

Final effect:

Blocked Information Page

In the/shell/yf domain, hold down F5 and refresh all the time. A few seconds later, the unblocking information and the unblocking time will be displayed. As long as a user's access to/shell/yf exceeds the normal frequency, the service will shut down the user for a period of time and automatically unseal the user upon expiration.

 

2. System Principle

Operating System: CentOS 6.5 x86_64
Development language: Bash Shell Scripts
Web server: Apache Httpd

2.1 custom log:/etc/httpd/logs/yfddos_log

 

Add the following two lines to the httpd. conf log parameters:

LogFormat "%a \"%U\" %{local}p %D %{%s}t " yfddosCustomLog logs/yfddos_log yfddos

Next we will focus on analyzing the log file/etc/httpd/logs/yfddos_log.

LogFormat "%a \"%U\" %{local}p %D %{%s}t " yfddos

Explanation:

% A-> User's IP % U-> request URL, but does not contain query string (The URL path requested, not including any query string .) % {local} p-> The server port requested by the user (usually 80) % D-> The request consumes the server in microseconds (The time taken to serve the request, in microseconds .) % {% s} t-> the timestamp value when the server receives this request (seconds since 00:00:00 UTC)

Example:

192.168.31.1 "/shell/yf" 80 118231 1417164313

Host whose IP address is 192.168.31.1. When the timestamp is 1417164313,/shell/yf is accessed and the server port 80 provides services to it, which takes 118231 microseconds.

Or:Host with IP address 192.168.31.1, accessed/shell/yf at 16:45:13 on and provided services to the server through port 80, which took 0.118231 seconds.

The reasons for not using the officially defined logs in httpd. conf are as follows:

-A user's access log record can be controlled within 60 bytes, with a small amount of data for later analysis. The officially defined log is too bloated and affects the analysis speed-use a timestamp to mark the time, for later analysis, the log time parameter officially defined is a regular expression, which is not easy to directly process-the httpd log system itself sorts records from old to new, therefore, the timestamp of the/etc/httpd/logs/yfddos_log log entries is also sorted from small to large, and the data records are more distinctive.

2.2 yfddosd blacklist File Format

Blacklist File Format

Yfddosd blacklist file/etc/yfddos/web-yf-search. B format is as follows:

# ip    add-stamp  rmv-stamp1.2.3.4 1416046335 14160463951.2.3.5 1416046336 14160463961.2.3.6 1416046339 1416046399

Each row is a blacklist entry. The first entry indicates:

IP Address: 1.2.3.4 Start Time: Timestamp 1416046335, that is, 18:12:15 End Time: Timestamp 1416046395, that is, 18:13:15

Intuitive meaning:

IP Address: 1.2.3.4. It has been banned for one minute since 18:12:15, January 15,. it is automatically unblocked at 18:13:15 January 15.

This file will be maintained and updated by the daemon.

2.3 daemon yfddosd: logic core of the Defense System

Schematic diagram of the daemon process

The yfddosd daemon is the core of the CC defense system.function analyze_and_insert_black()Is the core of yfddosd.

Configuration parameters of yfddosd:

Yfddos_blackfilePath = '/etc/yfddos/web-yf-search. B' yfddos _ accesslogPath = '/etc/httpd/logs/yfddos_log' function evaluate () {# analyze_and_insert_black (): # $1: max frequency (seems as abuse if above that) $2: blackip-ttl, time to live, unit is seconds (s) # $3: the access log $ {3} seconds before will be analyzed to generate the abuse ip lists that we will block # example: analyze_and_insert_black "limit" "ttl" "time" # example: analyze_and_insert_black "4" "10" "5" # analyze the user access logs in the past 5s. If someone visits the logs in the past 5s> = 4, the system adds them as resource abuse users. service blacklist # The Blacklist takes 10 s to apply. After 10 s, the system automatically deletes this blacklist entry and the Service continues to be open to it # global vars: # stamp logtmpfile yfddos_blackfilePath #......}

Functionanalyze_and_insert_blackThere are three input parameters:

Example:analyze_and_insert_black "4" "10" "5"

Explanation: analyze the user access logs in the/etc/httpd/logs/yfddos_log file in the past five seconds. If there is an IP address, the access volume is greater than or equal to 4 in the past five seconds, the daemon yfddosd will regard it as a resource abuse, and then add this IP address to the blacklist file/etc/yfddos/web-yf-search. B, the blacklist takes effect for 10 s, after 10 s, the daemon yfddosd deletes this blacklist entry.

Example:analyze_and_insert_black "150" "2700" "905"

Explanation: in the log file/etc/httpd/logs/yfddos_log, if there are IP addresses in the user access logs in the past 905s, the access volume is greater than or equal to 150, the daemon yfddosd will regard it as a resource abuse, and then add this IP address to the blacklist file/etc/yfddos/web-yf-search. B, the blacklist takes effect for 2700 s, after 2700s, the daemon yfddosd deletes this blacklist entry.

Note:analyze_and_insert_black "limit" "ttl" "time"

Explanation: in the log file/etc/httpd/logs/yfddos_log, if there is an IP address in the user access log in the past (time) s) s access volume> = limit, daemon yfddosd will regard it as a resource abuse, then this IP will be added to the blacklist file/etc/yfddos/web-yf-search. B, the valid time is (ttl) s. After (ttl) s, the daemon yfddosd automatically deletes this entry.

As can be seen from the above, the daemon yfddosd must complete at least three tasks:

-Analyze user access records within a specified time in the/etc/httpd/logs/yfddos_log log file-Add the IP address of the resource misuse owner to the file/etc/yfddos/web-yf-search. B, and set the block TTL parameter value-delete all expired entries in/etc/yfddos/web-yf-search. B in time

 

DaemonyfddosdHow to implement the above three Logics:

-Analysis of user access records within the specified time in the log file/etc/httpd/logs/yfddos_log: (1) extract the access log data of the past time seconds in/etc/httpd/logs/yfddos_log, and compress the time complexity of this operation to K * log2 (N) using the binary method, where N is the total number of log lines in/etc/httpd/logs/yfddos_log, and K is the time consumed for a test, which is generally within 1 ms. If there are 1048576 access records, this operation only requires 20*1 ms (2) to use the regular RE to perform secondary processing on the data, filter out the IP addresses of all users accessing the specified URL (this URL is the http service url to be defended. For example, in the system, the protected URL is/shell/yf, this service provides the search and get services for visitors), and uses sort and uniq to process these IP addresses again, in order to calculate the number of visits per IP address and sort the High and Low-Add the IP address of resource misuse to the file/etc/yfddos/web-yf-search. B, and set the block TTL parameter value to update all the IP addresses that exceed the limit threshold to the blacklist file/etc/yfddos/web-yf-search. B, the blocking time for each blacklist entry is ttl seconds-Remove all expired entries in/etc/yfddos/web-yf-search. B in time to traverse all blacklist entries in/etc/yfddos/web-yf-search. B, delete all expired entries one by one based on the current Timestamp

The following is the pseudo code of the yfddosd state machine daemon: (some processing details are omitted)

# Init and FSM start work... counter = 0 while truedo sleep 5 counter = counter + 1 delete obsolete items # delete all expired entries in/etc/yfddos/web-yf-search. B if # every 5 seconds: 5s then analyze_and_insert_black "6" "10" "5" # analyze users accessed in the past 5s if their access volume is greater than or equal to 6, the system will regard it as resource abuse # then add it the service blacklist takes 10 s. After 10 s, the daemon process automatically deletes this ip blacklist entry fi if # every 5*3 seconds: 15 s then analyze_and_insert_black "14" "45" "15" fi if # every 5*3*4 + 5 seconds: 65 s then analyze_and_insert_black "40" "840" "65" fi if # every 5*3*4*3*5 + 5 seconds: 905 s: 15 min then analyze_and_insert_black "150" "2700" 905 "fi if # every 5*3*4*3*5*4 + 5 seconds: 3605 s: 1 h then analyze_and_insert_black "300" "7200" 3605 "fi if # every 5*3*4*3*5*4*3 + 5 seconds: 10805 s: 3 h thenanalyze_and_insert_black "400" "21600" 10805 "if # perform then only once a day from-every day # backup log fi fidone

 

The defender should adjust the parameter values for each detection time point (blocking time ttl and determining threshold limit) to adjust the response time when the system responds to CC attacks.

3. Source Code

#! /Bin/bash #################################### # vim/usr/local/bin/yfddosd. sh: ##################################### nohup bash/ usr/local/bin/yfddosd. sh &> "/etc/yfddos/" "yfddosd-log-'date + % Y-% m-% d '"&########### ######################### yfddos daemonmkdir/etc/yfddosyfddos_blackfilePath = '/etc/yfddos/web-yf-search. B 'yfddos _ accesslogPath = '/etc/httpd/logs/yfddos_log' ### refresh tlllogtmpfile = 'mktem P 'Stamp = 'date + % s' touch "$ yfddos_blackfilePath" if grep-po' [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/nullthen cat "$ yfddos_blackfilePath" | while read I do deadstamp = 'echo "$ I" | grep-Po' [0-9] + $ ''if [" $ stamp "-le" $ deadstamp "] then echo" $ I ">" $ logtmpfile "fi donefichmod o + r" $ logtmpfile "mv-f" $ logtmpfile "" $ yfddos_blackfilePath "if! Grep-Po '[0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/nullthen echo '2017. limit limit 255 0 0 '> "$ yfddos_blackfilePath" Limit unction analyze_and_insert_black () {# analyze_and_insert_black () :#$ 1: max frequency (seems as abuse if above that) $2: blackip-ttl, time to live, unit is seconds (s) # $3: the access log $ {3} seconds before will be analyzed to generate the abuse ip lists that we w Ill block # example: analyze_and_insert_black "limit" "ttl" "time" # example: analyze_and_insert_black "4" "10" "5" # analyze the user access logs in the past 5s. If someone visits the logs in the past 5s> = 4, the system adds them as resource abuse users. service blacklist # The Blacklist takes 10 s to apply. After 10 s, the system automatically deletes this blacklist entry and the Service continues to be open to it # global vars: # stamp logtmpfile yfddos_blackfilePath local threshold = "$1" local ttl = "$2" local stamp_pre = "$3" local I = 0 local num = "" local fre = 0 local ip = 0 local localbuf = 0 loc Al linenum = 0 local deadstamp = 0 stamp_pre = "$ (stamp-stamp_pre )) "# initialize local temp = 0 local yf_x = '1' local yf_y = 'cat" $ logtmpfile "| wc-l' if [" $ yf_y "-le" 1"] then yf_y = 1 fi local yf_ I = $ (yf_x + yf_y) /2 )) temp = 'cat "$ logtmpfile" | wc-l 'if ["$ temp"-gt "0"] then temp = 'sed-n' $ P' "$ logtmpfile" | grep-Po '[0-9] + $ ''if [" $ temp "-lt" $ stamp_pre "] then num =" "else while true # use binary search quick analysis Access log do temp = 'sed-n "$ {yf_x} p" "$ logtmpfile" | grep-po' [0-9] + $ ''if [" $ temp" -ge "$ stamp_pre"] then break fi if ["$ (yf_y-yf_x )) "-le" 1 "] then yf_x =" $ yf_y "break fi temp = 'sed-n" $ {yf_ I} p "" $ logtmpfile "| grep-Po '[0 -9] + $ ''if [" $ temp "-lt" $ stamp_pre "] then yf_x =" $ yf_ I "yf_y =" $ yf_y "yf_ I =" $ ((( yf_x + yf_y) /2) "continue fi yf_x =" $ yf_x "yf_y =" $ yf_ I "yf_ I =" $ (yf_x + yf_y)/2) "continue don E temp = 'sed-n "$ {yf_x} p" "$ logtmpfile" | grep-po' [0-9] + $ ''if [" $ temp "-ge "$ stamp_pre"] then num = "$ yf_x" else num = "" fi if [-n "$ num"] then sed-n "$ {num }, \ $ p "" $ logtmpfile "| grep-Po '^ [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '| sort-n | uniq-c | sort-rn | while read I do fre = 'echo "$ I" | grep-po' [0- 9] + '| head-1' ip = 'echo "$ I" | grep-po' [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + ''if ["$ fre"-g E "$ threshold"] then # insert illegal ips: cat "$ yfddos_blackfilePath" # ip add-stamp rmv-stamp #1.2.3.4 1416046335 1416046395 temp = 'grep-Pn "$ {ip //. /\\.} "" $ yfddos_blackfilePath "'if [-n" $ temp "] then linenum = 'echo" $ temp "| grep-po' ^ [0-9] +' | head- 1 'deststamp' = 'echo "$ temp" | grep-po' [0-9] + $ '| sort-rn | head-1' if ["$ (stamp + ttl )) "-gt" $ deadstamp "] then sed-I" $ {linenum} s /. */$ {Ip }$ {stamp} $ (stamp + ttl )) /g "" $ yfddos_blackfilePath "fi else sed-I" \ $ a $ {ip }$ {stamp} $ (stamp + ttl )) "" $ yfddos_blackfilePath "fi else break fi done fi} # init and yfddosd's FSM start work... counter = 0 while truedo sleep 5 counter = $ (counter + 1) echo-n 'date + % Y-% m-% d \ % H: % M: % S' "counter $ {counter}:" 'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "echo-n" refresh tll: "'cat/proc/uptime | Grep-Po '[0-9 \.] + '| head-1' "" ### refresh tll # refresh ttl: analyze file: "$ yfddos_blackfilePath" if some items 'ttl has been reach the date, we will remove it and open service to the ip had been banned before. # insert illegal ips: cat "$ yfddos_blackfilePath" # ip add-stamp rmv-stamp #1.2.3.4 1416046335 1416046395 # sed-I "/^. * $ (stamp-5) $/d;/^. * $ (stamp-4) $/d;/^. * $ (stamp-3) $/d;/^. * $ (Stamp-2) $/d;/^. * $ (stamp-1) $/d;/^. * $ (stamp) $/d; /^ $/d "" $ yfddos_blackfilePath "logtmpfile = 'mktemp 'stamp = 'date + % s' touch" $ yfddos_blackfilePath "if grep-Po' [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/null then cat "$ yfddos_blackfilePath" | while read I do deadstamp = 'echo "$ I" | grep-Po '[0-9] + $ ''if [" $ stamp "-le" $ deadstamp "] then echo" $ I ">" $ logtmpfile "fi done Fi chmod o + r "$ logtmpfile" mv-f "$ logtmpfile" "$ yfddos_blackfilePath" if! Grep-Po '[0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/null then echo '2017. 255.255.255 0 0'> "$ yfddos_blackfilePath" fi logtmpfile = 'mktemp 'stamp = 'date + % s' cat "$ yfddos_accesslogPath" | grep-p' "/shell/yf" '> "$ logtmpfile" if true # every 5 seconds: 5S then echo-n "analyze_and_insert_black 6 10 5:" 'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "" # analyze yfddos log: Analyze_and_insert_black () $1: max frequency (seems as abuse if above that) $2: blackip-ttl $3: the access log $ {3} seconds before will be analyzed to generate the abuse ips that we will block analyze_and_insert_black "6" "10" "5" # analyze users who have accessed the service in the past 5s if a user's access volume is greater than or equal to 6, the system will regard it as a resource abuse user and add it to the service blacklist. The function time is 10 s. After 10 s, the daemon process will automatically delete this ip blacklist entry fi ["$ (counter % (3 ))) "-eq" 0 "] # every 5*3 seconds: 15 s then echo-n" analy Ze_and_insert_black 14 45 15: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "10" "45" 15 "fi if [" $ (counter % (3*4 + 1 ))) "-eq" 0 "] # every 5*3*4 + 5 seconds: 65 s then echo-n" analyze_and_insert_black 40 840 65: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black" limit" "Ttl" "time" analyze_and_insert_black "25" "840" "65" fi if ["$ (counter % (3*4*3*5 + 1 ))) "-eq" 0 "] # every 5*3*4*3*5 + 5 seconds: 905 s: 15 min then echo-n" analyze_and_insert_black 150 2700 905: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "150" "2700" 905 "fi if [" $ (counter % (3*4*3*5*4 + 1 ))) "-eq" 0 "] # e Very 5*3*4*3*5*4 + 5 seconds: 3605 s: 1 h then echo-n "analyze_and_insert_black 300 7200 3605: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "300" "7200" 3605 "fi if [" $ (counter % (3*4*3*5*4*4*3 + 1 ))) "-eq" 0 "] # every 5*3*4*3*5*4*3 + 5 seconds: 10805 s: 3 h then echo-n "analyze_and_insert_black 400 21600 10805: "'Cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "400" 21600 "" 10805 "####" $ {yfddos_accesslogPath} "backup: back up logs once every day from to if ["'date + % H'"-le "5"] &! [-F "$ {yfddos_accesslogPath}-'date + % Y-% m-% d'"] then service httpd stop mv "$ {yfddos_accesslogPath}" "$ {yfddos_accesslogPath }- 'date + % Y-% m-% d' "service httpd start fi rm-fr" $ logtmpfile "echo" sleep: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "done