Is it an attack?

System log: kernel: possible SYN flooding on port 80. Sending cookies.

Log: <P_UnixNet.h: 279 (check_throttle_warning)> too restart connections, throttling

The lvs System Log prompts that the backend server service is unavailable, and the number of connections to a single server through ipvsadmin is very high, up to +

The above logs indicate that there are many network connections, and the most likely cause is syn semi-open link attacks.

So what is syn semi-open link attack?

First, understand the tcp three-way handshake process

 

In linux, there are a total of 12 Possible statuses. the first 11 are described based on the three-way handshake established by the TCP connection and the four-way handshake process of the TCP connection disconnection.

1) LISTEN: first, The server needs to open a socket for listening. The status is LISTEN./* The socket is listening for incoming connections. LISTEN for connection requests from The remote TCP port */

2) SYN_SENT: the client calls connect through the application to perform active open. therefore, the client tcp sends a SYN request to establish a connection. then the status is set to SYN_SENT. /* The socket is actively attempting to establish a connection. wait for the matched connection request after sending the Connection Request */

3) SYN_RECV: the server should issue ACK to confirm the SYN of the client and send a SYN to the client. then the status is set to SYN_RECV/* A connection request has been received ed from the network. wait for confirmation of the connection request after receiving and sending a connection request */

4) ESTABLISHED: indicates an open connection. Both parties can interact with each other or already interact with each other. /* The socket has an established connection. indicates an opened connection. data can be transmitted to The user */

5) FIN_WAIT1: the application of the active close end calls close, so TCP sends a FIN request to actively close the connection, and then enters the FIN_WAIT1 status. /* The socket is closed, and the connection is shutting down. wait for the confirmation of the remote TCP connection interruption request or the previous connection interruption request */

6) CLOSE_WAIT: After the passive close end TCP receives the FIN, it sends an ACK to respond to the FIN request (its receipt is also passed to the upper-layer application as the file Terminator ), and enter CLOSE_WAIT. /* The remote end has shut down, waiting for the socket to close. waiting for a connection interruption request from a local user */

7) FIN_WAIT2: After the active shutdown end receives ACK, it enters FIN-WAIT-2. /* Connection is closed, and the socket is waiting for a shutdown from the remote end. wait for connection interruption requests from remote TCP */

8) LAST_ACK: After the end is passively closed for a period of time, the application receiving the file Terminator will call CLOSE to CLOSE the connection. As a result, TCP also sends a FIN, waiting for the ACK of the other party. the LAST-ACK. /* The remote end has shut down, and the socket is closed. waiting for acknowledgement. WAIT for confirmation of the original connection interruption request sent to remote TCP */9) and TIME_WAIT: After receiving the FIN from the active closing end, TCP sends the ACK packet and enters the TIME-WAIT Status. /* The socket is waiting after close to handle packets still in the network. Wait for enough time to ensure that The remote TCP receives the confirmation of The connection interruption request */

10) CLOSING: rare./* Both sockets are shut down but we still don't have all our data sent. Wait for remote TCP to confirm the connection interruption */

11) CLOSED: After the passively closed end receives the ACK packet, it enters the CLOSED status. Connection ended./* The socket is not being used. No connection status */

12) UNKNOWN: UNKNOWN Socket status. /* The state of the socket is unknown .*/

SYN: (synchronous Sequence number, Synchronize Sequence Numbers) this flag is valid only when three handshakes are used to establish a TCP connection. Indicates a New TCP connection request.

ACK: (confirmation Number, Acknowledgement Number) indicates the confirmation of TCP requests. It also prompts that the peer system has successfully received all data.

FIN: (end sign, FINish) is used to end a TCP session. However, the corresponding port is still open and is ready to receive subsequent data.

Half-Open link attacks use the three-way handshake vulnerability of ipv4 tcp to exhaust the resources of the attacked party (the CPU is full or the memory is insufficient) only twice)

Supplement:
Statistics link status: netstat-nat | awk '{print awk $ NF}' | sort | uniq-c | sort-n
Count the total number of links for each IP80 port and sort them in descending order: netstat-nat | grep ": 80" | awk '{print $5}' | awk-F: '{print $1}' | sort | uniq-c | sort-n
Syncookies are used to prevent Synflood attacks. The principles are as follows:
SYN cookies allow the server to process new SYN requests when the semi-connection queue is full. When the semi-connection queue is full, SYN cookies do not discard SYN requests, instead, it performs hash operations on the source IP address, source port number, initial client serial number received, and other security values, and encrypts them to obtain the initial serial number of the server, which is called a cookie. After the server sends a SYN + ACK packet whose initial serial number is cookie, it releases the allocated connection request block. If the ACK package of the client is received, the server reduces the ACK serial number of the client by 1 and compares it with the hash value obtained by the preceding element. If it is equal, the three-way handshake is directly completed, create a new connection. The core of the SYN cookies mechanism is to avoid a large number of useless connection request blocks caused by attacks, resulting in memory depletion and failure to process normal connection requests.

There must be many links to abnormal Dos attacks. Is it Dos attacks if the link is exhausted?
Though further analysis, it is found that the number of logs with normal access to ts is small (xxxx.com and xxxx.cn), but the number of logs without xxxx is relatively large. By analyzing non-xxxx logs, it is found that, many other accesses, such as multi-play account scan requests and Youku search requests, are clearly exploited by attackers to act as proxies to do bad things, one of these requests caught my attention. Because the CDN cache server does not limit the CDN domain name, this will cause Cache Server loopback requests, which will eventually exhaust the server's network connection and fail to provide services normally.
What did this accident bring.
1. Adjust the ts configuration, only match the xxxx domain name, and tell them that this is not a free proxy. I only work for xxxx.
2. Security reinforcement: the Internet CDN environment, unlike self-built IDCs, does not have a firewall and the switch does not allow acl policies. all protection measures can only be performed by the host, popular host protection tools include iptables and selinux.
In response to this problem, iptables has been adjusted. In the past, ts only allowed access from tcp port 80 to the mac address matching the Server Load balancer, and only allowed access from Server Load balancer to port 80 of ts.
Selinux is very powerful. This is a brief introduction (currently we are enabling the permissive mode ):

SELinux, short for Security Enhanced Linux (Security Enhanced Linux), is an implementation of Mandatory Access Control (Mandatory Access Control System, the purpose is to explicitly specify the resources (files, network ports, etc.) that a process can access ).

The purpose of the mandatory access control system is to enhance the system's ability to defend against 0-Day attacks (Attacks implemented by exploiting undisclosed vulnerabilities. Therefore, it is not a substitute for the network firewall or ACL, and is not used repeatedly.

For example, Apache on the system is found to have a vulnerability that allows a remote user to access sensitive files on the system (such as/etc/passwd to obtain existing users in the system ), however, the Apache update Patch for fixing this security vulnerability has not been released yet. In this case, SELinux can mitigate the vulnerability. Because/etc/passwd does not have the Apache access tag, Apache's access to/etc/passwd will be blocked by SELinux.

Compared with other mandatory access control systems, SELinux has the following advantages:

The control policy is queryable rather than invisible to the program.
You can hot change the policy without restarting or stopping the service.
The policy can be used to control process initialization, inheritance, and program execution.
The control scope covers the file system, directory, file, file start descriptor, port, message interface, and network interface.

Protection is often accompanied by performance issues,
The performance of iptables mainly depends on the complexity of the policy. In principle, the policy should be simple and effective. Currently, because the switch cannot implement the acl Policy, it can only be implemented through iptables. If a performance problem occurs, you can only choose to increase machine performance. After all, security is always the first.
Selinux has been compared by some authority. Enabling SELinux only reduces the system performance by about 5% in a few cases.
3. If syncookie is enabled for common ddos attacks, linux itself has a certain degree of fire prevention capability. If it is a higher level ddos attack, it should be done through the hardware traffic cleaning device.