[Linux network programming notes] TCP short connections generate a large number of time_wait, which makes it impossible to create new TCP connections and solutions-Basics

Recently, I encountered an online alarm: a large number of time_wait servers make it unable to establish a new HTTP connection with downstream modules. During the solution, I checked the classic textbooks and technologies.ArticleTo enhance the understanding of TCP network problems. This is recorded as a note.
Note: This article mainly introduces the many basic knowledge involved in TCP programming. The solution to the new connection failure caused by time_wait in the actual project will be given in the next note.

1. Actual Problems
Preliminary check found that when a TCP connection cannot be created externally, a large number of TCP connections in the time_wait status exist on the online server (up to 10 million + single-host connections at a time, the time_wait generated by the module that caused the alarm is about 2 W), which makes it unable to establish a new TCP connection with the downstream module.
Time_wait involves status migration during TCP release connection and the impact of specific socket APIs on TCP status. These concepts are gradually introduced below.

2. TCP status migration
connection-oriented TCP protocol requires that a TCP connection be established before each peer communication. The connection can be abstracted as a 4-tuples (four-tuple, (local_ip, local_port, remote_ip, remote_port). These four elements uniquely represent a TCP connection.
1) TCP connection establishment
the TCP connection establishment process is usually called "three-way handshake ),

 It can be explained as follows:
A. the client sends SYN to the server and specifies that the initial package number (sequence number) is J;
B. the server sends its own SYN Packet and indicates that the initial package number is K. At the same time, ackj + 1 is returned for synj of the client (note: J + 1 indicates that the server expects the next package from the client to be in the order of J + 1 );
C. After the client receives the SYN + ACK from the server, it sends ackk + 1. TCP is established successfully.
In fact, during the three handshakes established by TCP, we also need to use the SYN packet to determine their MSS, timestamp, and other parameters. This involves the details of the Protocol. This article aims to give a reference and will not start any further.

2) tcpconnection termination
In response to the three handshakes that establish the connection, when releasing a TCP connection, it needs to go through four steps (also known as "four waves"), as shown in:

you can explain the following:
. A connected party first calls close () to initiate active close. This API will prompt the TCP transport layer to send a FIN packet to remotepeer, this package indicates that the application initiating the active close will not send data (Note: here, the promise of "no longer send data" is from the application layer perspective. At the TCP transmission layer, or send the unsent data in the kernel TCP send buffer corresponding to the application to the link ). After the remote peer receives the fin, it needs to complete the passive close (passive close), which is divided into two steps:
B. first, at the TCP transmission layer, the ACK packet is first sent to the FIN packet of the other party (the primary ACK packet order is based on the FIN packet order of the other Party plus 1);
C. then, the application at the application layer receives the peer's EOF (end-of-file, and the peer's Fin package is used as the EOF application to the application at the application layer), knowing that this connection will no longer have data from the peer, therefore, close () is also called to close the connection, which will prompt the TCP transport layer to send fin.
D. The peer initiating the active shutdown sends an ACK packet after receiving the remote peer's Fin. At this point, the TCP connection is closed.
Note 1: either party of the TCP connection can first call close () to initiate an active shutdown and use the client to automatically initiate a shutdown, instead, the client can only initiate automatic shutdown.
NOTE 2: the preceding TCP connection creation/release process is described, if you do not consider the protocol details such as retransmission and congestion control caused by various reasons, you can view various tcp rfc documents, such as TCP rfc793.

 

3) TCP statetransition dimo-
The above describes the process of establishing and releasing a connection over TCP. Here we will give a general description of the migration process of the TCP state machine. The TCP state machine migration diagram described in TCP rfc793 is extracted as follows (referenced here ):

The TCP state machine contains 11 statuses, And the statuses are migrated under various socket APIs drivers. Although this figure looks complicated, for those who have some experience in TCP network programming, it is easy to understand. For more information about the migration process, see section 2.6 of Linux Network Programming volume1.

 

3. time_wait status
After the above preparations, we will finally discuss the content related to the topic of this article. Pai_^
The TCP status migration diagram shows that only the party that calls close () to initiate an active shutdown will enter the time_wait status, in addition, it is required to enter (the three State Migration lines shown in the lower left corner of the figure will eventually enter this State to return to the initial closed state ).
You can also see that the TCP connection that enters the time_wait status must pass through 2msl to return to the initial status, where MSL refers to Max
Segment lifetime, that is, the maximum lifetime of a data packet in the network. An appropriate MSL value should be specified for each TCP protocol implementation method. For example, the recommended value of rfc1122 is 2 minutes. For example, TCP implementation of the Berkeley system usually selects 30 seconds as the MSL value. This means that the typical duration of time_wait is 1-4 minutes.
There are two main reasons for the time_wait status:
1) Reliable release for TCP full-duplex connections
Refer to the TCP release connection mentioned earlier in this article, and assume that the ACK (the last packet of 4 interactions) sent by the party initiating active close (the client in the figure) is lost in the network, because of the TCP retransmission mechanism, the party executing passiveclose (the server in the figure) needs to resend its fin before the fin arrives at the client (the client is the active close initiator, the client must maintain the connection status (although it has called close). Specifically, the resources corresponding to the TCP connection (local_ip, local_port) cannot be immediately released or reassigned. The TCP connection can be restored to the initial closed State only when the fin resent by the romete peer is reached and the Ack is re-sent by the client. If the activeclose side does not enter time_wait to maintain its connection status, when the passive Close side resends the fin, the TCP transport layer of the active Close side will respond to the other side with the RST packet, this will be considered an error by the other party (in fact, this is a normal connection close process, not an exception ).
2) in order to make the old data packet disappear due to network expiration
To illustrate this problem, we first assume that the TCP protocol does not have a time_wait status limit, and then assume that there is a TCP connection: (local_ip, local_port, remote_ip, remote_port). For some reason, close the connection first, and then quickly create a new connection with the same four tuples. As described earlier in this article, TCP connections are uniquely identified by tuples. Therefore, in our assumptions, the TCP protocol stack cannot distinguish the two TCP connections. In its view, this is basically the same connection. The process of first releasing and then establishing in the middle is "imperceptible" to it. This may happen when the data sent by the local peer in the previous TCP connection reaches the remote peer, the TCP transmission layer of the remot peer receives and transmits the normal data of the current TCP connection to the application layer (in fact, in the scenario we assume, before the old data arrives at the remote peer, the old connection is disconnected and a new TCP connection consisting of the same four tuples has been established. Therefore, these old data should not be passed up to the application layer), which leads to data disorder and various unpredictable strange phenomena. As a reliable transmission protocol, TCP must consider and avoid this situation at the protocol level, which is the 2nd reason for the existence of time_wait status.
Specifically, after the local peer actively calls close, the TCP connection enters the time_wait state, and the TCP connection in this state cannot immediately establish a new connection with the same four elements, that is, the local port occupied by the initiator of active close cannot be reassigned during time_wait. Because the time_wait status lasts for 2msl, this ensures that the old data packets in the duplex link of the old TCP connection disappear due to expiration (beyond MSL, you can use the same four tuples to create a new connection without data disorder between the first and second connections.

4. socket API: Close () and Shutdown ()
As shown in the previous figure, for a TCP connection, the party that calls close () first enters the time_wait status, in addition, there are some details about close.
the default action of calling close () for a TCP socket is to mark the socket as closed and return it to the process of calling the API immediately. In this case, the socket FD cannot be used by the process at the application layer, that is, it cannot be used as a parameter for read or write. From the perspective of the transport layer, TCP will try to send the backlog of data in the current send buffer to the link, and then initiate the TCP four waves to completely close the TCP connection.
calling close () is a normal way to close the TCP connection, but there are two restrictions in this method, which is exactly why introduces Shutdown:
1) Close () reduces the reference count of socket FD by 1. Only when the reference count of socket FD is reduced to 0, the TCP transport layer will initiate four handshakes to actually close the connection. Shutdown can directly initiate the four handshakes required to close the connection without being limited by the reference count.
2) Close () terminates the TCP duplex link. Due to the full duplex feature of the TCP connection, the local peer may not send data to the remote peer, and the remote peer may have data to send, in this case, if the local peer wants to notify the remote peer that it will not send data but will continue to receive data, close () will not work, and Shutdown () this task can be completed.

For details about the call methods of close () and Shutdown (), refer to man.

The above is the basic knowledge required to analyze and solve the problem "too many time_wait Connections cannot be established externally. The next note will introduce the specific solution to this problem based on this article. Pai_^

[References]
1. Linux Network Programming Volume 1. Chapter 2 & Chapter 4
2. tcp rfc 793
3. online document: TCP statetransition dimo-

====================== EOF ============================