Diagnosis and handling of TCP Flood attacks (SYN Flood)

Attack principle:

SYN Flood is one of the most popular DoS (Denial of Service Attack) and DDoS (Distributed Denial of Service Attack) methods, send a large number of forged TCP connection requests, and send the first handshake packet (SYN Packet) of a massive request connection from a counterfeit IP or IP segment ), the attacked server responds to the second handshake packet (SYN + ACK packet). because the other party is a fake IP address, the other party will never receive the packet and will not respond to the third handshake packet. As a result, the attacked server maintains a large number of "semi-Connections" in the SYN_RECV state and retries to respond to the second handshake packet five times by default, which is full of TCP waiting connection queues, when resources are exhausted (the CPU is full or the memory is insufficient), normal service requests cannot be connected.

Diagnosis:

When we saw a sharp drop in the business curve, we checked the machine and DNS and found that the external machine was slow in response, high CPU load, slow ssh Login, and even some machines could not log on. We checked the system syslog:

# Tail-f/var/log/messages
Apr 18 11:21:56 web5 kernel: possible SYN flooding on port 80. Sending cookies.

Check that the number of connections increases, and the number of SYN_RECV connections is particularly high:

# Netstat-n | awk '/^ tcp/{++ S [$ NF]} END {for (a in S) print, S [a]} 'time _ WAIT 16855CLOSE_WAIT 21SYN_SENT 99FIN_WAIT1 229FIN_WAIT2 113 ESTABLISHED 8358SYN_RECV 48965 CLOSING 3LAST_ACK 313 check the number of connections normally as follows: # netstat-n | awk '/^ tcp/{++ S [$ NF]} END {for (a in S) print, S [a]} 'time _ WAIT 42349CLOSE_WAIT 1SYN_SENT 4FIN_WAIT1 298FIN_WAIT2 33 ESTABLISHED 12775SYN_RECV 259 CLOSING 6LAST_ACK 432

Emergency Response

According to the characteristics of the Peer IP address viewed by netstat:
# Netstat-na | grep SYN_RECV | more

Use iptables to temporarily block the IP or IP number segment of the largest suspected attack, such as the counterfeit 173. *. *. * segment no. is used to launch an attack. In the short term, 173 is disabled. *. *. * This large segment (Be sure not to block your local IP address !)
# Iptables-a input-s 173.0.0.0/8-p tcp-dport 80-j DROP

Analyze the evidence that has just been retained, analyze the business, and use iptables to unseal normal ip addresses and subnet segments in Section 173. In this way, the emergency response is easy to get hurt by mistake. It is not an ideal way to log on to the server through ssh even if an error is blocked.

Tcp_synack_retries = 0 is the key, indicating that after the second handshake packet (SYN + ACK packet) is returned to the Client IP address, if the third handshake packet (ACK packet) is not received, speed up the recycling of "semi-Connections" and do not consume resources.

 

If you do not modify this parameter to simulate an attack, port 80 attacked in 10 seconds will not be able to serve the service, and it is difficult for the machine to log on via ssh; run the netstat-na | grep SYN_RECV command to detect that "semi-connection" is held for 180 seconds;

Modify this parameter to 0, and then simulate the attack. After 10 minutes, port 80 attacked will be able to provide services, and the response will be slower, but ssh may not be able to log on; detection of "semi-connection" is released after being held for 3 seconds.

Side effect of modifying this parameter to 0: if the other party does not receive the second handshake packet when the network condition is poor, the connection to the server may fail, but for general websites, the user can refresh the page once. These can be captured during peak hours or when the network condition is poor.

Based on previous packet capture experience, this is rare, but for the sake of security, this parameter can be temporarily enabled only when it is under tcp flood attacks.

The default value of tcp_synack_retries is 5, indicating that the task is resold for 5 times, waiting for 30 ~ 40 seconds, that is, the "semi-connection" is held for about 180 seconds by default. Explanation:

The tcp_synack_retries setting tells the kernel how many times to retransmit the SYN, ACK reply
An SYN request. In other words, this tells the system how many times to try to establish a passive
TCP connection that was started by another host.
This variable takes an integer value, but shocould under no circumstances be larger than 255 for
Same reasons as for the tcp_syn_retries variable. Each retransmission will take aproximately 30-40
Seconds. The default value of the tcp_synack_retries variable is 5, and hence the default timeout
Of passive TCP connections is aproximately 180 seconds.

 

You can change tcp_synack_retries to 0 because the client also has the tcp_syn_retries parameter. The default value is 5. Even if the server does not resend the SYN + ACK packet, the client resends the SYN handshake packet. Explanation:

The tcp_syn_retries variable tells the kernel how many times to try to retransmit the initial SYN
Packet for an active TCP connection attempt.
This variable takes an integer value, but shocould not be set higher than 255 since each
Retransmission will consume huge amounts of time as well as some amounts of bandwidth. Each
Connection retransmission takes aproximately 30-40 seconds. The default setting is 5, which
Wocould lead to an aproximate of 180 seconds delay before the connection times out.

 

Second ParameterNet. ipv4.tcp _ max_syn_backlog = 200000It is also important that the specific number of values is limited by the memory.

In the following configuration, the first parameter is the most important parameter, the second parameter is the auxiliary parameter, and other parameters have other functions:

# Vi/etc/sysctl. conf

Net. ipv4.tcp _ synack_retries = 0 # length of the semi-connection queue. net. ipv4.tcp _ max_syn_backlog = 200000 # maximum number of file handles allowed by the system, because the connection occupies the file handle fs. file-max = 819200 # used to handle sudden large concurrent connect requests. net. core. somaxconn = 65536 # maximum TCP Data receiving buffer (in bytes) net. core. rmem_max = 1024123000 # maximum TCP data sending buffer (in bytes) net. core. wmem_max = 16777216 # maximum number of packets allowed to be sent to the queue when the network device receives packets at a rate faster than the kernel processes these packets. net. core. netdev_max_backlog = 165536 # port allocation range when the local machine actively connects to other machines. net. ipv4.ip _ local_port_range = 1000065535 #...... Omit other ......

 

Make the configuration take effect:
# Sysctl-p

Note: do not enable the following parameters when facing the internet. Because the side effects are obvious, please google for specific reasons. If you have already opened it, explicitly change it to 0, and then execute sysctl-p to close it. As a result of the experiment, a large number of connections in the TIME_WAIT status have little impact on the system:

 

# When the semi-connection queue overflows, send syncookies to the other party. It is unnecessary to increase the size of the semi-connection queue. net. ipv4.tcp _ syncookies = 0 # connection reuse function in TIME_WAIT status. net. ipv4.tcp _ tw_reuse = 0 # timestamp option, with the preceding net. ipv4.tcp _ tw_reuse parameter works with net. ipv4.tcp _ timestamps = 0 # connection recovery function in TIME_WAIT status. net. ipv4.tcp _ tw_recycle = 0