Automatic Defense System for CC attacks-principles and implementation

Automatic Defense System for CC attacks-principles and implementation
0x00 system effect this DDOS Application Layer defense system has been deployed on the http://www.bkjia.com site (if access fails, please directly access the server in China http: // 121.42.45.55 for online testing ).
The defense system is at the application layer, which effectively prevents the abuse of server resources by illegal users:

As long as it sends high-frequency and application-layer requests to achieve a large amount of system resource consumption attacks, it can effectively defend against attacks.

The basic idea of its implementation is:

Periodically analyze the request frequency of all access users in the past period of time. If the request frequency is higher than the specified threshold, the user is identified as a resource abuse user. After the validity period, the defense system automatically unblocks it.

Online performance testing:


Go to http://www.yfdc.org-> click the upper right side Online queryIn /shell/yfDomain, /shell/yfYes bash scriptsWrite Dynamics web-cgiProgram, the user submits information each time, this program will execute some server-side query operations, and then return the data processing to the client.
To prevent unauthorized users from accessing this program frequently and affect the access of other normal users, some protection measures are required.
Final effect:


Blocked Information Page

In the/shell/yf domain, hold down F5 and refresh all the time. A few seconds later, the unblocking information and the unblocking time will be displayed. As long as a user's access to/shell/yf exceeds the normal frequency, the service will shut down the user for a period of time and automatically unseal the user upon expiration.

 
0x01 system principles
Operating System: CentOS 6.5 x86_64  Development language: Bash Shell Scripts  Web server: Apache Httpd


(This figure shows the system structure of the bird's eye chart that can be saved locally and enlarged for viewing)
2.1 custom log:/etc/httpd/logs/yfddos_log


(Custom log file format)


Add the following two lines to the httpd. conf log parameters:

LogFormat "%a \"%U\" %{local}p %D %{%s}t " yfddosCustomLog logs/yfddos_log yfddos

Next we will focus on analyzing the log file/etc/httpd/logs/yfddos_log.

LogFormat "%a \"%U\" %{local}p %D %{%s}t " yfddos

Explanation:

% A-> User's IP % U-> request URL, but does not contain query string (The URL path requested, not including any query string .) % {local} p-> The server port requested by the user (usually 80) % D-> The request consumes the server in microseconds (The time taken to serve the request, in microseconds .) % {% s} t-> the timestamp value when the server receives this request (seconds since 00:00:00 UTC)

Example:

192.168.31.1 "/shell/yf" 80 118231 1417164313

Host whose IP address is 192.168.31.1. When the timestamp is 1417164313,/shell/yf is accessed and the server port 80 provides services to it, which takes 118231 microseconds.
Or: The host with the IP address 192.168.31.1 was accessed by/shell/yf at 16:45:13 and provided services to the host through port 80 of the server, which took 0.118231 seconds.
The reasons for not using the officially defined logs in httpd. conf are as follows:

-A user's access log record can be controlled within 60 bytes, with a small amount of data for later analysis. The officially defined log is too bloated and affects the analysis speed-use a timestamp to mark the time, for later analysis, the log time parameter officially defined is a regular expression, which is not easy to directly process-the httpd log system itself sorts records from old to new, therefore, the timestamp of the/etc/httpd/logs/yfddos_log log entries is also sorted from small to large, and the data records are more distinctive than 2.2 yfddosd blacklist file formats.


Blacklist File Format


Yfddosd blacklist file/etc/yfddos/web-yf-search. B format is as follows:

# ip    add-stamp  rmv-stamp1.2.3.4 1416046335 14160463951.2.3.5 1416046336 14160463961.2.3.6 1416046339 1416046399

Each row is a blacklist entry. The first entry indicates:

IP Address: 1.2.3.4 Start Time: Timestamp 1416046335, that is, 18:12:15 End Time: Timestamp 1416046395, that is, 18:13:15

Intuitive meaning:
IP Address: 1.2.3.4. It has been banned for one minute since 18:12:15, January 15,. it is automatically unblocked at 18:13:15 January 15.
This file will be maintained and updated by the daemon.
2.3 daemon yfddosd: logic core of the Defense System



Schematic diagram of the daemon process
The yfddosd daemon is the core of the CC defense system. function analyze_and_insert_black()Is the core of yfddosd.
Configuration parameters of yfddosd:

Yfddos_blackfilePath = '/etc/yfddos/web-yf-search. B' yfddos _ accesslogPath = '/etc/httpd/logs/yfddos_log' function evaluate () {# analyze_and_insert_black (): # $1: max frequency (seems as abuse if above that) $2: blackip-ttl, time to live, unit is seconds (s) # $3: the access log $ {3} seconds before will be analyzed to generate the abuse ip lists that we will block # example: analyze_and_insert_black "limit" "ttl" "time" # example: analyze_and_insert_black "4" "10" "5" # analyze the user access logs in the past 5s. If someone visits the logs in the past 5s> = 4, the system adds them as resource abuse users. service blacklist # The Blacklist takes 10 s to apply. After 10 s, the system automatically deletes this blacklist entry and the Service continues to be open to it # global vars: # stamp logtmpfile yfddos_blackfilePath #......}

Function analyze_and_insert_blackThere are three input parameters:

Example: analyze_and_insert_black "4" "10" "5"

Explanation: analyze the user access logs in the/etc/httpd/logs/yfddos_log file in the past five seconds. If there is an IP address, the access volume is greater than or equal to 4 in the past five seconds, the daemon yfddosd will regard it as a resource abuse, and then add this IP address to the blacklist file/etc/yfddos/web-yf-search. B, the blacklist takes effect for 10 s, after 10 s, the daemon yfddosd deletes this blacklist entry.

Example: analyze_and_insert_black "150" "2700" "905"

Explanation: in the log file/etc/httpd/logs/yfddos_log, if there are IP addresses in the user access logs in the past 905s, the access volume is greater than or equal to 150, the daemon yfddosd will regard it as a resource abuse, and then add this IP address to the blacklist file/etc/yfddos/web-yf-search. B, the blacklist takes effect for 2700 s, after 2700s, the daemon yfddosd deletes this blacklist entry.

Note: analyze_and_insert_black "limit" "ttl" "time"

Explanation: in the log file/etc/httpd/logs/yfddos_log, if there is an IP address in the user access log in the past (time) s) s access volume> = limit, daemon yfddosd will regard it as a resource abuse, then this IP will be added to the blacklist file/etc/yfddos/web-yf-search. B, the valid time is (ttl) s. After (ttl) s, the daemon yfddosd automatically deletes this entry.

As can be seen from the above, the daemon yfddosd must complete at least three tasks:


Analyze user access records in the specified time in the/etc/httpd/logs/yfddos_log Log File
Add the IP address of the resource misuse owner to the file/etc/yfddos/web-yf-search. B and set the block TTL parameter value
Delete all expired entries in/etc/yfddos/web-yf-search. B in time

DaemonyfddosdHow to implement the above three Logics:

Analyze the user access records within the specified time in the/etc/httpd/logs/yfddos_log log file:

(1) extract access log data from/etc/httpd/logs/yfddos_log in the past time seconds, and use the binary method to compress the time complexity of this operation to K * log2 (N) or less, where N is the total number of log lines in/etc/httpd/logs/yfddos_log, and K is the time consumed for a test, which is generally within 1 ms. If there are 1048576 access records, this operation only requires 20*1 ms (2) to use the regular RE to perform secondary processing on the data, filters out all the user IP addresses that access the specified URL (this URL is the http service url to be defended, for example, in the http://www.yfdc.org system, what is defended against is/shell/yf, this service provides the search and get services for visitors), and uses sort and uniq to process these IP addresses again, so as to count the number of visits to each IP address and sort them in descending order.

Add the IP address of the resource misuse owner to the file/etc/yfddos/web-yf-search. B and set the block TTL parameter value

Update all IP addresses that exceed the limit threshold to the blacklist file/etc/yfddos/web-yf-search. B, with each blacklist entry blocked for ttl seconds

Delete all expired entries in/etc/yfddos/web-yf-search. B in time

Traverse all black list entries in/etc/yfddos/web-yf-search. B and delete all expired entries one by one based on the current Timestamp

 

The following is the pseudo code of the yfddosd state machine daemon: (some processing details are omitted)

# Init and FSM start work... counter = 0 while truedo sleep 5 counter = counter + 1 delete obsolete items # delete all expired entries in/etc/yfddos/web-yf-search. B if # every 5 seconds: 5s then analyze_and_insert_black "6" "10" "5" # analyze users accessed in the past 5s if their access volume is greater than or equal to 6, the system will regard it as resource abuse # then add it the service blacklist takes 10 s. After 10 s, the daemon process automatically deletes this ip blacklist entry fi if # every 5*3 seconds: 15 s then analyze_and_insert_black "14" "45" "15" fi if # every 5*3*4 + 5 seconds: 65 s then analyze_and_insert_black "40" "840" "65" fi if # every 5*3*4*3*5 + 5 seconds: 905 s: 15 min then analyze_and_insert_black "150" "2700" 905 "fi if # every 5*3*4*3*5*4 + 5 seconds: 3605 s: 1 h then analyze_and_insert_black "300" "7200" 3605 "fi if # every 5*3*4*3*5*4*3 + 5 seconds: 10805 s: 3 h then analyze_and_insert_black "400" "21600" "10805" if # perform then only once a day from-every day # backup log fi fidone

The defender should adjust the parameter values for each detection time point (blocking time ttl and determining threshold limit) to adjust the response time when the system responds to CC attacks.

0x02 source code

##################################### Vim/usr /local/bin/yfddosd. sh: ##################################### nohup bash/ usr/local/bin/yfddosd. sh &> "/etc/yfddos/" "yfddosd-log-'date + % Y-% m-% d '"&########### ######################### yfddos daemonmkdir/etc/yfddosyfddos_blackfilePath = '/etc/yfddos/web-yf-search. B 'yfddos _ accesslogPath = '/etc/httpd/logs/yfddos_log' ### refresh tlllogtmpfile = 'mktemp 'stamp = 'Date + % s' touch "$ yfddos_blackfilePath" if grep-po' [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/nullthen cat "$ yfddos_blackfilePath" | while read I do deadstamp = 'echo "$ I" | grep-Po' [0-9] + $ ''if [" $ stamp "-le" $ deadstamp "] then echo" $ I ">" $ logtmpfile "fi donefichmod o + r" $ logtmpfile "mv-f" $ logtmpfile "" $ yfddos_blackfilePath "if! Grep-Po '[0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/nullthen echo '2017. limit limit 255 0 0 '> "$ yfddos_blackfilePath" fi function analyze_and_insert_black () {# analyze_and_insert_black (): # $1: max frequency (seems as abuse if above that) $2: blackip-ttl, time to live, unit is seconds (s) # $3: the access log $ {3} seconds before will be analyzed to generate the abuse ip lists that we Will block # example: analyze_and_insert_black "limit" "ttl" "time" # example: analyze_and_insert_black "4" "10" "5" # analyze the user access logs in the past 5s. If someone visits the logs in the past 5s> = 4, the system adds them as resource abuse users. service blacklist # The Blacklist takes 10 s to apply. After 10 s, the system automatically deletes this blacklist entry and the Service continues to be open to it # global vars: # stamp logtmpfile yfddos_blackfilePath local threshold = "$1" local ttl = "$2" local stamp_pre = "$3" local I = 0 local num = "" local fre = 0 local ip = 0 local localbuf = 0 lo Cal linenum = 0 local deadstamp = 0 stamp_pre = "$ (stamp-stamp_pre )) "# initialize local temp = 0 local yf_x = '1' local yf_y = 'cat" $ logtmpfile "| wc-l' if [" $ yf_y "-le" 1"] then yf_y = 1 fi local yf_ I = $ (yf_x + yf_y) /2 )) temp = 'cat "$ logtmpfile" | wc-l 'if ["$ temp"-gt "0"] then temp = 'sed-n' $ P' "$ logtmpfile" | grep-Po '[0-9] + $ ''if [" $ temp "-lt" $ stamp_pre "] then num =" "else while true # use binary search quick Method Analyze the access log do temp = 'sed-n "$ {yf_x} p" "$ logtmpfile" | grep-po' [0-9] + $ ''if [" $ temp "-ge" $ stamp_pre "] then break fi if [" $ (yf_y-yf_x )) "-le" 1 "] then yf_x =" $ yf_y "break fi temp = 'sed-n" $ {yf_ I} p "" $ logtmpfile "| grep-Po '[0 -9] + $ ''if [" $ temp "-lt" $ stamp_pre "] then yf_x =" $ yf_ I "yf_y =" $ yf_y "yf_ I =" $ ((( yf_x + yf_y) /2) "continue fi yf_x =" $ yf_x "yf_y =" $ yf_ I "yf_ I =" $ (yf_x + yf_y)/2) "continue do Ne temp = 'sed-n "$ {yf_x} p" "$ logtmpfile" | grep-po' [0-9] + $ ''if [" $ temp "-ge "$ stamp_pre"] then num = "$ yf_x" else num = "" fi if [-n "$ num"] then sed-n "$ {num }, \ $ p "" $ logtmpfile "| grep-Po '^ [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '| sort-n | uniq-c | sort-rn | while read I do fre = 'echo "$ I" | grep-po' [0- 9] + '| head-1' ip = 'echo "$ I" | grep-po' [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + ''if ["$ fre "- Ge "$ threshold"] then # insert illegal ips: cat "$ yfddos_blackfilePath" # ip add-stamp rmv-stamp #1.2.3.4 1416046335 1416046395 temp = 'grep-Pn "$ {ip //. /\\.} "" $ yfddos_blackfilePath "'if [-n" $ temp "] then linenum = 'echo" $ temp "| grep-po' ^ [0-9] +' | head- 1 'deststamp' = 'echo "$ temp" | grep-po' [0-9] + $ '| sort-rn | head-1' if ["$ (stamp + ttl )) "-gt" $ deadstamp "] then sed-I" $ {linenum} s /. */$ {Ip} $ {stamp} $ (stamp + ttl )) /g "" $ yfddos_blackfilePath "fi else sed-I" \ $ a $ {ip }$ {stamp} $ (stamp + ttl )) "" $ yfddos_blackfilePath "fi else break fi done fi} # init and yfddosd's FSM start work... counter = 0 while truedo sleep 5 counter = $ (counter + 1) echo-n 'date + % Y-% m-% d \ % H: % M: % S' "counter $ {counter}:" 'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "echo-n" refresh tll: "'cat/proc/upti Me | grep-po' [0-9 \.] + '| head-1' "" ### refresh tll # refresh ttl: analyze file: "$ yfddos_blackfilePath" if some items 'ttl has been reach the date, we will remove it and open service to the ip had been banned before. # insert illegal ips: cat "$ yfddos_blackfilePath" # ip add-stamp rmv-stamp #1.2.3.4 1416046335 1416046395 # sed-I "/^. * $ (stamp-5) $/d;/^. * $ (stamp-4) $/d;/^. * $ (stamp-3) $/d;/^. * $ (Stamp-2) $/d;/^. * $ (stamp-1) $/d;/^. * $ (stamp) $/d; /^ $/d "" $ yfddos_blackfilePath "logtmpfile = 'mktemp 'stamp = 'date + % s' touch" $ yfddos_blackfilePath "if grep-Po' [0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/null then cat "$ yfddos_blackfilePath" | while read I do deadstamp = 'echo "$ I" | grep-Po '[0-9] + $ ''if [" $ stamp "-le" $ deadstamp "] then echo" $ I ">" $ logtmpfile "fi d One fi chmod o + r "$ logtmpfile" mv-f "$ logtmpfile" "$ yfddos_blackfilePath" if! Grep-Po '[0-9] + \. [0-9] + \. [0-9] + \. [0-9] + '"$ yfddos_blackfilePath" &>/dev/null then echo '2017. 255.255.255 0 0'> "$ yfddos_blackfilePath" fi logtmpfile = 'mktemp 'stamp = 'date + % s' cat "$ yfddos_accesslogPath" | grep-p' "/shell/yf" '> "$ logtmpfile" if true # every 5 seconds: 5S then echo-n "analyze_and_insert_black 6 10 5:" 'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "" # analyze yfddos log: Analyze_and_insert_black () $1: max frequency (seems as abuse if above that) $2: blackip-ttl $3: the access log $ {3} seconds before will be analyzed to generate the abuse ips that we will block analyze_and_insert_black "6" "10" "5" # analyze users who have accessed the service in the past 5s if a user's access volume is greater than or equal to 6, the system will regard it as a resource abuse user and add it to the service blacklist. The function time is 10 s. After 10 s, the daemon process will automatically delete this ip blacklist entry fi ["$ (counter % (3 ))) "-eq" 0 "] # every 5*3 seconds: 15 s then echo-n" analy Ze_and_insert_black 14 45 15: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "10" "45" 15 "fi if [" $ (counter % (3*4 + 1 ))) "-eq" 0 "] # every 5*3*4 + 5 seconds: 65 s then echo-n" analyze_and_insert_black 40 840 65: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black" limit" "Ttl" "time" analyze_and_insert_black "25" "840" "65" fi if ["$ (counter % (3*4*3*5 + 1 ))) "-eq" 0 "] # every 5*3*4*3*5 + 5 seconds: 905 s: 15 min then echo-n" analyze_and_insert_black 150 2700 905: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "150" "2700" 905 "fi if [" $ (counter % (3*4*3*5*4 + 1 ))) "-eq" 0 "] # e Very 5*3*4*3*5*4 + 5 seconds: 3605 s: 1 h then echo-n "analyze_and_insert_black 300 7200 3605: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "300" "7200" 3605 "fi if [" $ (counter % (3*4*3*5*4*4*3 + 1 ))) "-eq" 0 "] # every 5*3*4*3*5*4*3 + 5 seconds: 10805 s: 3 h then echo-n "analyze_and_insert_black 400 21600 10805: "'Cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "# example: analyze_and_insert_black "limit" "ttl" "time" analyze_and_insert_black "400" 21600 "" 10805 "####" $ {yfddos_accesslogPath} "backup: back up logs once every day from to if ["'date + % H'"-le "5"] &! [-F "$ {yfddos_accesslogPath}-'date + % Y-% m-% d'"] then service httpd stop mv "$ {yfddos_accesslogPath}" "$ {yfddos_accesslogPath }- 'date + % Y-% m-% d' "service httpd start fi rm-fr" $ logtmpfile "echo" sleep: "'cat/proc/uptime | grep-po' [0-9 \.] + '| head-1' "done