Another time_wait.

So the topic is because a long time ago I have written an article introducing Time_wait, but at that time basically belong to a little, and did not go into the ins and outs of the problem, it happened that this period of time repeatedly asked about the related issues, let me feel the need for a comprehensive summary, in case of a rainy day.

Before the discussion, you can take the server at hand to touch the mapping, remember "ss" than "netstat" faster:

Shell> Ss-ant | awk '    nr>1 {++s[$1]} END {for (k in s) print K,s[k]} '

If you just want to check the number of time_wait alone, it can be simpler:

Shell> Cat/proc/net/sockstat

I guess you must be shocked by the huge number of time_wait Internet connections! In my personal experience, for a busy Web server, the total number of time_wait network connections is likely to reach tens of thousands of, or even a hundred thousand of, if the primary connection is primarily short. Although a time_wait network connection consumes only a port, a bit of memory, but the Jiabuzhu cardinality is large, so this is always a need to face the problem.

Why is there a time_wait?

TCP requires a handshake when establishing a connection, and, in the same vein, a handshake when the connection is closed. To more visually illustrate the process of closing the handshake when connecting, we refer to the example in "the TCP/IP guide":

TCP Close

Because the TCP connection is bidirectional, each of the two directions needs to be closed when the connection is closed. The one party that first sends the FIN package performs the active shutdown, and the one on the back of the fin package executes the passive shutdown. The active closed party enters the TIME_WAIT state, and the MSL duration is two times longer in this state.

A little bit of MSL's knowledge: MSL refers to the maximum lifetime of a message segment, and if the message segment is active on the network, the MSL time is not received and is discarded. With respect to the size of the MSL, the recommendations in the RFC 793 protocol are two minutes, but actually different operating systems may have different settings, as in Linux, usually half a minute, two times the MSL is a minute, or 60 seconds, and the value is hardcoded in the kernel, This means that you cannot modify the kernel unless you recompile it:

#define Tcp_timewait_len (60*HZ)

If the number of connections per second is 1000, then a minute can generate 60,000 time_wait.

Why does the active closed party not go directly into the closed state, but enter the TIME_WAIT state, and stay two times the MSL length? This is because TCP is a reliable protocol built on unreliable networks. Example: The active shut-off party receives the fin packet sent by the passive closed party, responds to the ACK packet, and enters the time_wait state simultaneously, but because of network reasons, the ACK packet sent by the active shutdown party is likely to delay, thereby triggering the passive connection party to retransmit the fin packet. In extreme cases, this one goes back, which is twice times the MSL duration. If the active closed party skips the time_wait directly into the closed, or the MSL is less than twice times longer than the time_wait stay, then a problem similar to the following may occur when the passive closed party has previously issued a delay packet:

• The old TCP connection no longer exists, and the system can only return the RST package at this time

• A new TCP connection is set up, and the delay packet may interfere with the new connection

In either case, TCP is no longer reliable, so there is a need for the time_wait state to exist.

How to control the number of time_wait?

From the previous description we can draw the conclusion that time_wait this thing is not possible, but too much may be a nuisance. Let's see what we can do to control the number of time_wait, here are just some of the usual methods, and some other methods such as so_linger are too biased and skip the door.

Ip_conntrack: As the name implies is tracking connections. Once this module is activated, you can find many settings in the system parameters to control the network connection status timeout, which naturally includes time_wait:

shell> modprobe ip_conntrackshell> sysctl net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait

We can try to narrow down its settings, such as 10 seconds, or even a second, depending on how much of the network is appropriate, and of course you can refer to the relevant case. However, as far as my personal opinion is concerned, Ip_conntrack introduces more problems than the solution, such as the performance will be greatly reduced, so it is not recommended.

tcp_tw_recycle: As the name implies is recycling time_wait connection. It can be said that this kernel parameter has become a popular handling time_wait of the balm, if you search for TIME_WAIT solution on the network, nine to Ten will recommend setting it, but there is a hidden trap:

When multiple clients are networked through NAT and interacting with the server, the server sees the same IP, which means that these clients are actually equivalent to one on the server side, but due to the possible differences in timestamps of these clients, there may be a time stamp disorder from the point of view of the service side. Packets that result in a small timestamp are discarded. Reference: Tcp_tw_recycle and tcp_timestamps cause a connect failure issue.

Tcp_tw_reuse: As the name implies is the multiplexing time_wait connection. When creating a new connection, consider reusing the corresponding time_wait connection if possible. It is generally thought that "tcp_tw_reuse" is safer than "tcp_tw_recycle", because one time_wait creation time must be more than a second to be reused, and two dozen the time stamp of the connection is incremented. This is said in the official documentation: if it is safe from a protocol perspective, then it can be used. This is diplomatic! In my opinion, if the network is more stable, such as the intranet connection, then you can try to use.

However, it is important to note where to use, since we want to reuse the connection, then of course it should be used in the initiator of the connection, but not in the connected side of the use. For example: The client initiates an HTTP request to the server, and the server responds with an active shutdown of the connection, so the time_wait is left on the server, which is invalid with "tcp_tw_reuse" because the server is connected, so there is no multiplexing connection. Let us extend a point of view, for example, the server is PHP, it queries the other MySQL server, and then actively disconnect, so time_wait fell on the PHP side, in such cases, the use of "tcp_tw_ reuse"is valid because PHP is the client in relation to MySQL, and it is the initiator of the connection, so you can reuse the connection.

Note: If you use Tcp_tw_reuse, activate Tcp_timestamps, otherwise it will be invalid.

tcp_max_tw_buckets: As the name implies, controls the total number of time_wait. Official website The document says this option is only to prevent some simple Dos attacks, usually do not artificially reduce it. If it is reduced, then the system will delete the extra time_wait, the log will show: "tcp:time wait bucket table overflow".

Need to remind everyone is extremes meet, once saw someone put "tcp_max_tw_buckets" set to 0, that is to say completely abandoned time_wait, this is some adventure, with a go proverb: into the boundary should be slow.

...

Sometimes, if we look at the problem from a different angle, we can get the effect of four or two dials. The example mentioned earlier: The client initiates an HTTP request to the server, and the server actively shuts down the connection after the service side responds, so time_wait is left on the server. The key here is to actively close the connection is the service side! When the TCP connection is closed, the first shot of the party is doomed to escape the fate of the time_wait, to apply a word: My sorrow left to myself, your beauty let you take away. If the client is controllable, then open the keepalive on the server, and let the client actively shut down the connection as far as possible, so that the problem is resolved.

Reference Documentation:

1. TCP Short Connection time_wait problem solving method Daquan (1)--strategically advantageous position

2. TCP Short Connection time_wait problem solving method Daquan (2)--so_linger

3. TCP Short Connection time_wait problem solving method Daquan (3)--tcp_tw_recycle

4. TCP Short Connection time_wait problem solving method Daquan (4)--tcp_tw_reuse

5. TCP Short Connection time_wait problem solving method Daquan (5)--tcp_max_tw_buckets

Another time_wait.