Ngxin prevents concurrency (whitelist, with proxy on the front end)

Ngxin prevents concurrency (whitelist, with proxy on the front end)

Nginx has two modules to control the "quantity" and "Speed" of access. Simply put, it controls the maximum number of accesses you have at the same time and the maximum number of accesses you can access per second, you cannot have too many concurrent accesses at the same time, nor can it be too fast, otherwise it will be "no harm ".

HttpLimitZoneModule limits the number of concurrent accesses at the same time

HttpLimitReqModule restricts data access, up to several requests per second

Limit_conn_zone Syntax: limit_conn_zone $ variable zone = name: size; default value: none configuration segment: http this command describes the session Status storage area. The current number of connections is saved in the key status. The key value can be any non-null value of a specific variable (null values will not be considered ). $ Variable defines the key, zone = name defines the region name, which will be used by the limit_conn command later. Size defines the size of memory shared by each key. For example:
Limit_conn_zone $ binary_remote_addr zone = addr: 10 m;

Note: the IP address of the client is used as the key. Note that the $ binary_remote_addr variable is used, instead of the $ remote_addr variable. $ Remote_addr variables are 7 to 15 bytes in length, while the storage status occupies 32 or 64 bytes in 32-bit platforms and 64 bytes in 64-bit platforms. The length of the $ binary_remote_addr variable is fixed to 4 bytes. The storage status occupies 32 bytes or 64 bytes on the 32-bit platform and 64 bytes on the 64-bit platform. 1 m shared space can save 32 thousand 32-bit States and 16 thousand 64-bit states. If the shared memory space is exhausted, the server will return a 503 (Service Temporarily Unavailable) Error for all subsequent requests. The limit_zone command is equivalent to the limit_conn_zone command. If it has been discarded, it will not be explained.

Limit_conn_log_level Syntax: limit_conn_log_level info | notice | warn | default error value: error configuration segment: http, server, and location. When the maximum number of connections is reached, the log level is recorded.

Limit_conn Syntax: limit_conn zone_name number default value: none configuration segment: http, server, location specifies the maximum number of simultaneous connections for each given key value. If this number is exceeded, 503 (Service Temporarily Unavailable) is returned) error. For example:

limit_conn_zone $binary_remote_addr zone=addr:10m;server {    location /www.bkjia.com/ {        limit_conn addr 1;    }}

Only one connection is allowed at a time for the same IP address. When multiple limit_conn commands are configured, the limit on the number of connections takes effect. For example, the following configuration limits not only the number of connections from a single IP Address Source, but also the total number of connections from a single virtual server:

limit_conn_zone $binary_remote_addr zone=perip:10m;limit_conn_zone $server_name zone=perserver:10m;server {    limit_conn perip 10;    limit_conn perserver 100;}

[Warning] The limit_conn command can be inherited from the upper-level.
Limit_conn_status Syntax: limit_conn_status code; default value: limit_conn_status 503; configuration segment: http, server, location, which is introduced in version 1.3.15. Specifies the status code returned when the limit is exceeded. The default value is 503.

Limit_rate Syntax: limit_rate default value: 0 configuration segment: http, server, location, if in location for each connection speed limit. The unit of the parameter rate is byte/second. If it is set to 0, the speed limit is disabled. Limit by connection speed rather than by IP address. Therefore, if two connections are enabled for a client at the same time, the overall rate of the client is twice the value set by this command.

Limit_req_zone Syntax: limit_req_zone $ variable zone = name: size rate = rate; default value: none configuration segment: http sets a shared memory limit domain to save key value status parameters. In particular, the number of requests that exceed the current limit is saved. The key value is the specified variable (the null value is not calculated ). For example
Limit_req_zone $ binary_remote_addr zone = one: 10 m rate = 1r/s;

Note: The region name is one and the size is 10 MB. The average request frequency cannot exceed once per second. The key value is the Client IP address. With the $ binary_remote_addr variable, you can reduce the size of each state record to 64 bytes, so that the 1 MB memory can save about 10 thousand 64-byte records. If the storage space of the restricted domain is exhausted, the server will return the 503 (Service Temporarily Unavailable) Error for all subsequent requests. The speed can be set to the number of requests processed per second and the number of requests processed per minute. The value must be an integer. Therefore, if you need to specify to process less than one request per second, process one request in 2 seconds, you can use "30r/m ".

Limit_req_log_level Syntax: limit_req_log_level info | notice | warn | error; default value: limit_req_log_level error; configuration segment: http, server, location set the log level you want, when the server rejects or delays processing requests due to high frequency, you can write down logs of the corresponding level. The Log Level of the delayed record is one lower than that of the rejected log level. For example, if "limit_req_log_level notice" is set, the delayed log level is info.

Limit_req_status Syntax: limit_req_status code; default value: limit_req_status 503; configuration segment: http, server, location this command is introduced in version 1.3.15. Sets the response status code for a request rejection.

Limit_req Syntax: limit_req zone = name [burst = number] [nodelay]; default value:-configuration segment: http, server, location sets the corresponding shared memory limit domain and the maximum number of requests allowed to be processed. If the request frequency exceeds the value configured in the restricted domain, request processing will be delayed, so all requests are processed at the defined frequency. Requests that exceed the frequency limit will be delayed until the number of delayed requests exceeds the defined threshold, the request will be terminated and the 503 (Service Temporarily Unavailable) error will be returned. The default value of this threshold is 0. For example:

limit_req_zone $binary_remote_addr zone=limit_com:10m rate=1r/s;server {    location /www.bkjia.com/ {        limit_req zone=limit_com burst=5;    }}

Limit an average of no more than one request per second, and the number of requests allowed to exceed the frequency limit cannot exceed 5. If you do not want the request to be delayed, you can use the nodelay parameter, for example:

Limit_req zone = ttlsa_com burst = 5 nodelay;
-------- Configuration example --------------------------------------------

# Your IP address $ limit is used as the Key. Each IP Address can have up to 50 concurrent connections.

# How many thousands of connections do you want to destroy me? If there are more than 50 connections, error 503 is directly returned to you and your request is not processed.

However, this is handled by ngin and will not affect WEB applications such as backend tomcat. If the nginx Nic traffic is congested and the pressure on a single server is insufficient, you can find another solution. It will be explained later
Limit_req_zone $ limit zone = tlcy_com: 10 m rate = 10r/s; limit_req_log_level info; limit_conn_zone $ limit zone = addr: 10 m; limit_conn_log_level info;

# Your IP address $ limit is used as the Key. Each IP address processes 10 requests per second.

# If you want to use a program to refresh me several hundred times per second, you will not be able to deal with it any more soon. You will be directly returned with the 503 Error
# Server Configuration
 

http｛....    limit_req_zone  $limit zone=tlcy_com:10m rate=10r/s;  limit_req_log_level  info;  limit_conn_zone  $limit zone=addr:10m;  limit_conn_log_level info;  server  {    listen      80;    server_name  www.bkjia.com;    if ($http_user_agent ~* LWP::Simple|BBBike|wget|Sosospider|YodaoBot) {   return 403;  } ##    root  /data/www/;##    index  hou.txt;   location /mp4/    {    if ($request_method !~ ^(GET|HEAD|POST)$ ) {            return 444;   }                    }    location / {     if ($request_method !~ ^(GET|HEAD)$ ) {            return 444;        }             proxy_next_upstream http_502 http_504 error timeout invalid_header;            proxy_pass http://tlcy;            proxy_redirect    off;            proxy_set_header  Host $host;            proxy_set_header  X-Real-IP  $remote_addr;            proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;            allow all;                }

# Up to 5 queues. Because you process 10 requests and 5 queues per second, you can send up to 15 requests at a time,
If there are more, the error 503 is returned to you.

 limit_req zone=tlcy_com burst=5  nodelay;   limit_conn addr 10;      location ~* \.(gif|jpg|png|swf|flv)$ {     valid_referers none blocked www.bkjia.com ;     if ($invalid_referer) {     rewrite ^/ http://www.bkjia.com/403.html;   #return 404;  }   }   }

Transactions have two sides. Although the ngx_http_limit_conn_module can solve the current concurrency problems, it will introduce other problems. For example, if the frontend performs LVS or reverse generation, and the backend enables this module function, isn't it a very many 503 errors? In this way, you can enable this module on the front end or set the whitelist.
---------------------------- Whitelist settings -----------------------------------------

? # Configure http {.... geo $ white_ip {default 1; 127.0.0.1 0; 10.0.0.0/8 0;} # whitelist map $ white_ip $ limit {1 $ binary_remote_addr; 0 "";} limit_req_zone $ limit zone = tlcy_com: 10 m rate = 10r/s; limit info; limit_conn_zone $ limit zone = addr: 10 m; limit_conn_log_level info; server {listen 80; server_name www.hzcsky.com; if ($ http_user_agent ~ * LWP: Simple | BBBike | wget | Sosospider | YodaoBot) {return 403 ;## root/data/www/;## index hou.txt; location/mp4/{if ($ request_method! ~ ^ (GET | HEAD | POST) $) {return 444 ;}} location/{if ($ request_method !~ ^ (GET | HEAD) $) {return 444;} proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_pass http://tlcy ; Proxy_redirect off; proxy_set_header Host $ host; proxy_set_header X-Real-IP $ remote_addr; proxy_set_header X-Forwarded-For $ proxy_add_x_forwarded_for; allow all ;}

# Up to 5 queues. Because you process 10 requests and 5 queues per second, you can send up to 15 requests at a time,
If there are more, the error 503 is returned to you.

 limit_req zone=tlcy_com burst=5  nodelay;   limit_conn addr 10;      location ~* \.(gif|jpg|png|swf|flv)$ {     valid_referers none blocked www.bkjia.com ;     if ($invalid_referer) {     rewrite ^/ http://www.bkjia.com/403.html;   #return 404;  }   }   }

################# If a single nginx server cannot solve this problem, LVS or haproxy must be used for Layer 4 and nginx must be used to solve this problem, if the IP addresses of multiple ECs instances are all lvs ip addresses, it will be very troublesome.

(Here only show the results, do not understand the Http protocol, please Google or Wikipedia http://zh.wikipedia.org/zh-cn/X-Forwarded-For)

When a CDN or transparent Proxy Server transfers user requests to the backend server, the CDN Server adds a record to the Http header.
X-Forwarded-For: user IP address, proxy server IP Address

If there is more than one proxy server in the middle, this record will be like this after a multi-layer proxy is established in the middle of www.bzfshop.net.
X-Forwarded-For: user IP, proxy server 1-IP, proxy server 2-IP, proxy server 3-IP ,....

You can see that after multi-layer proxy, the real IP address of the user is in the first position, and the IP address of the intermediate proxy server will be followed by a string of IP addresses, from which the real IP address of the user is obtained, you can restrict the IP address,
 
Nginx Configuration

Obtain the original user address

Enable log display:
Log_format main '$ http_x_forwarded_for $ remote_addr--$ time_iso8601 "$ request_method $ scheme: // $ host $ request_uri $ server_protocol "$ status $ bytes_sent" $ http_referer "" $ http_user_agent "$ request_time $ upstream_cache_status: TCP ';
Map $ http_x_forwarded_for $ clientRealIp {
# Directly use remote_addr without using a proxy
"" $ Remote_addr;
# Use regular expression matching to obtain the user's original IP address from x_forwarded_for
# For example, X-Forwarded-For: 202.123.123.11, 208.22.22.234, 192.168.2.100 ,...
# Here, the first 202.123.123.11 is the user's real IP address, followed by the CDN Server
~ ^ (? PfirstAddr> [0-9 \.] + ),?. * $ FirstAddr;
}
 
# Using the map command, we created a variable $ clientRealIp for nginx, which is the real IP address of the original user,
# We can obtain the correct original IP address no matter whether the user accesses it directly or after a string of CDN accesses
 
Complete configuration example:

# Configure http {map $ http_x_forwarded_for $ limit {"" $ remote_addr ;~ ^ (? P [0-9 \.] + ),?. * $ FirstAddr;} # map $ white_ip $ limit {#1 $ clientRealIp; #0 "" ;#} limit_req_zone $ limit zone = tlcy_com: 10 m rate = 5r/s; limit_req_log_level info; limit_conn_zone $ limit zone = addr: 10 m; limit_conn_log_level info; server {listen 80; server_name www.bkjia.com; if ($ http_user_agent ~ * LWP: Simple | BBBike | wget | Sosospider | YodaoBot) {return 403 ;## root/data/www/;## index hou.txt; location/mp4/{if ($ request_method! ~ ^ (GET | HEAD | POST) $) {return 444 ;}} location/{if ($ request_method !~ ^ (GET | HEAD) $) {return 444;} proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_pass http://tlcy ; Proxy_redirect off; proxy_set_header Host $ host; proxy_set_header X-Real-IP $ remote_addr; proxy_set_header X-Forwarded-For $ scheme; allow all ;}## a maximum of five queues are supported, since 10 requests and 5 queues are processed per second, you can send up to 15 requests in one second. If there are more requests, the system returns the 503 error message limit_req zone = tlcy_com burst = 5 nodelay; limit_conn addr 10; location ~ * \. (Gif | jpg | png | swf | flv) $ {valid_referers none blocked www.bkjia.com; if ($ invalid_referer) {rewrite ^/ http://www.bkjia.com/403.html ; # Return 404 ;}}}

This is because the original IP address does not need to be set to a whitelist. You can directly access the nginx layer. The original IP address is not limited.

You need to properly seal the IP address and use the log with the script:
#! /Bin/bash
# Obtain the parameter $1 as the concurrency threshold. If it is left blank, the maximum concurrency of A Single IP address is 200 by default!

If [[-z $1]; then num = 200 else num = $ 1fi LOG =/root/log/nginx/sns_access.logSTATUS = 503 # function check () {iplist = 'cat $ LOG | grep-I $ STATUS | grep-I ", "| awk '{print $2}' | grep-v"-"| sed" s #, # g "| sort | uniq-cd | sort-rn | awk-v str = $ num' {if ($1> str) {print $2} ''if [[! -Z $ iplist]; then>/data/shell/black_ip.txt for black_ip in $ iplist do # The IP segment determination function has been canceled in whitelist filtering, you can modify the following code as needed # exclude_ip = 'echo $ black_ip | awk-F ". "'{print $1 ". "$2 ". "$3}'' # grep-q $ exclude_ip. /white_ip.txt grep-q $ black_ip/data/shell/white_ip.txt if [[$? -Eq 0]; then echo "$ black_ip (white_ip) ">>>/data/shell/black_ip.txt else echo $ black_ip>/data/shell/black_ip.txt # iptables-nL | grep $ black_ip | (iptables-I INPUT-s $ black_ip -j DROP & echo "$ black_ip 'date + % Y-% m-% H: % M: % S' ">/data/shell/denylog.txt) fi done # send an email if a single IP address whose concurrency exceeds the threshold # if ['cat. /sendmail '= 1]; then sendmsg; fi} function checka () {iplist = 'cat $ LOG | grep-I $ STATUS | grep- V "," | awk '{print $1}' | sort | uniq-cd | sort-rn | awk-v str = $ num' {if ($1> str) {print $2} ''if [[! -Z $ iplist]; then>/data/shell/black_ip.txt for black_ip in $ iplist do # The IP segment determination function has been canceled in whitelist filtering, you can modify the following code as needed # exclude_ip = 'echo $ black_ip | awk-F ". "'{print $1 ". "$2 ". "$3}'' # grep-q $ exclude_ip. /white_ip.txt grep-q $ black_ip/data/shell/white_ip.txt if [[$? -Eq 0]; then echo "$ black_ip (white_ip) ">>>/data/shell/black_ip.txt else echo $ black_ip>/data/shell/black_ip.txt # iptables-nL | grep $ black_ip | (iptables-I INPUT-s $ black_ip -j DROP & echo "$ black_ip 'date + % Y-% m-% H: % M: % S' ">/data/shell #/denylog.txt) fi done # send an email if a single IP address whose concurrency exceeds the threshold # if ['cat. /sendmail' = 1]; then sendmsg; fi} # function of sending an email: sendmsg () {netstat-nutlp | grep "sendmail"> /Dev/null 2> & 1 |/etc/init. d/sendmail start>/dev/null 2> & 1 echo-e "From: email address @ qq.com \ nTo: email address @ qq.com \ nSubject: Someone Attacking your system !! \ NIts Ip is ">. /message cat. /black_ip.txt>. /message/usr/sbin/sendmail-f email address @ qq.com-t email address @ qq.com-I/message>. /sendmail }## interval 10 s infinite loop check function # while true # do # check every 10 s, time can be customized as needed # sleep 10 # done #

Check # process IP addresses without proxies

Checka # process multi-layer IP addresses

We cut logs every 5 minutes. Therefore, check for unblocked IP addresses within five minutes, and write a scheduled task to reset IPtables within two hours.