In the morning, the site visit suddenly became very difficult, initially suspected that the machine room routing problem (once out), and the computer room after the replacement of the route, but the problem remains.
At noon, accidentally found a large number of links in the Syn_recv state, Google was suspected of SYN flood attack.
To query the number of connections in the SYN State, you can use this command:
Netstat-atn | Grep-c SYN
Query for the most connected IP:
Netstat-atn | grep "SYN" | awk ' {print $} ' | Awk-f ': ' {print $} ' | Sort-nr | uniq-c | Sort-nr | Head
A number of IP connections were found to be above 200 and are basically identified as SYN attacks.
And then they started to ban the IP by hand:
Iptables-a input-p tcp-s attacker Ip-dport 80-j REJECT
Because at work, there is no more time to deal with, only to ban some of the number of connections over 50 IP, barely maintain the site access.
9.19 a.m.
With the help of several friends, a script was changed to automate the above work. Every minute check the SYN state of the number of connections too high IP, and then ban off.
September 19
The script is very effective, except that the load is slightly higher, access is normal.
September 20
When the attacker discovers that SYN is invalid, it uses a join attack with the wait state instead. This is relatively good to do, as long as the above code in the SYN to change to "syn\\| Wait "can be. Such as:
Netstat-atn | Grep-c "syn\\| Wait "
September 22
As the attack intensified, it was possible that the attacker had invoked more machines to launch the attack. Server load up to 800-900,SSH completely unable to log in.
Notify the room to stop the telecommunications IP, the other side really do not know the machine's netcom IP. Telecom IP stopped, smoothly through the Netcom IP login shell.
There is no better way to change some shell kernel parameters, including enabling Syn cookies, and then change the automatic Ban IP script to more than 30 slow connections on ban.
But this is easy to manslaughter, such as some schools, companies and Internet café users, they are the same site IP.
After this change, and then reluctantly braced.
September 23
finally found a radical solution: use Nginx when Apache's reverse proxy.
Nginx is a Web server and reverse proxy developed by the old Mozie, putting it in front of Apache as a front-end Web server.
Because it itself has a mechanism for preventing slow connections (the principle is not to mention, Google a lot), so first of all the 80-port access to Nginx to deal with.
For pictures such as static files,/js/css by Nginx directly back, not to Apache.
And the PHP file is handed to Apache processing.
TIP: There is also a benefit, if the Nginx connection set to Keep-alive, and Apache set to close, overall efficiency will be much higher.
In this way, the effect is surprisingly good. The number of processes in Apache is reduced directly from 200+ to about 15, while the Nginx itself is only 2, and the system load drops directly.
And for slow connections, even a few IP wait status links up to 1000 + also have no impact on site access.
TIP: Remember to set Apache to only listen for 127.0.0.1 access, so as not to bypass the external direct nginx impact Apache.
Here's an article about this structure: http://kovyrin.net/2006/05/18/nginx-as-reverse-proxy/