Tcp_tw_reuse, tcp_tw_recycle use scenes and precautions

Linux time_wait related parameters:

Net.ipv4.tcp_tw_reuse = 0    means turn on reuse. Allows time-wait sockets to be reused for new TCP connections, which defaults to 0, which means that closing net.ipv4.tcp_tw_recycle = 0  means fast recycling of time-wait sockets on a TCP connection, which defaults to 0. Indicates off net.ipv4.tcp_fin_timeout =  If the socket is closed by the local side, this parameter determines the time it remains in the fin-wait-2 state (can be changed to 30, generally fin-wait-2 connection is very small)

Attention:

-Unlike Windows can modify the registry to modify the value of 2MSL, Linux is no way to modify the MSL, Tcp_fin_timeout is not 2MSL but fin-wait-2 state.

-Tcp_tw_reuse and SO_REUSEADDR are two totally different things.

1. tw_reuse,tw_recycle must be enabled when client and server timestamps are turned on (default on)

2. The Tw_reuse only works on the client and the client is recovered within 1s after opening

3. Tw_recycle to the client and the server at the same time, after opening in 3.5*rto recovery, RTO 200ms~ 120s time depending on the network conditions.

Intranet situation is slightly faster than tw_reuse, public network especially mobile network is much slower than Tw_reuse, the advantage is to be able to reclaim the number of time_wait on the service side

For client

1. As a client because of the port 65535 problem, time_out too much directly affect the processing power, open tw_reuse can be resolved, not recommended to open tw_recycle at the same time, not helpful.

2. tw_reuse help the client 1s to complete the connection recovery, the basic can achieve a single 6w/s request, you need to increase the number of IP.

3. If the intranet is measuring the scene, and the client does not need to receive the connection, Tw_recycle will have a little benefit.

4. Business can also be designed by the server to actively close the connection

For service-side

1. Invalid Open Tw_reuse

2. On-line environment tw_recycle do not open

After the server is in a NAT load, or after the client is in NAT (this is something, the basic corporate Home network goes NAT);

Public NETWORK service may cause partial connection failure, the intranet can be opened as appropriate;

Like my company external services are placed behind the load, the load will be emptied timestamp, OK, even if you open it does not work.

3. What about server time_wait high?

Unlike the client has port restrictions, processing a large number of Time_wait Linux has been optimized well, each in time_wait state the connection memory consumption is very low,

It can also be configured with a maximum limit of tcp_max_tw_buckets = 262144 , and modern machines generally do not lack this memory.

Below is the HTTP short connection service of a peak 1w request per second, long-term tw_buckets overflow state,

Tw_socket_tcp occupies 70M because the business simple service consumes CPU 200% running very stably.

Slabtop

262230 251461 95% 0.25K 17482 69928K tw_sock_tcp

ss-stotal:259 (kernel 494) TCP:   262419 (estab 113, closed 262143, orphaned 156, Synrecv 0, timewait 262143/0), ports 80Transport Total     IP        ipv6*         494       -         -        RAW       1         1         0        UDP       0         0         0        TCP       276       276       0        INET      277       277       0        FRAG      0         0         0    

The only thing that bothers me is:

Overflow error in System log always swipe the screen, maybe the buckets is a little bit bigger.

Tcp:time Wait Bucket table overflow
Tcp:time Wait Bucket table overflow
Tcp:time Wait Bucket table overflow
Tcp:time Wait Bucket table overflow
Tcp:time Wait Bucket table overflow

5. Business can also be designed by the client to actively close the connection

Principle Analysis

1. MSL origin

Initiates a connection the closing party replies to the last fin's ACK, to avoid the ACK received by the other side, the re-issued or still in the middle route of the fin to kill the new connection, wait for a 2msl,4min.

That is, the person who closes the connection has a time_wait problem, and the party that is closed has no such problem.

2. Reuse, Recycle

By timestamp the increment to differentiate whether a new connection, the new connection timestamp larger, then the small timestamp fin will not fin off the new connection.

3. Reuse

With timestamp increment, client and server can handle outofbind fin package

4. Recycle

For the service side, the same src IP, may be a lot of machines after NAT, these machines timestamp increment is not guaranteed, the server will reject the non-incremental request connection.

The details have to read the TCP protocol stack source.

It is recommended to read the following references:

Coping with the TCP time-waitstate on busy Linux servers

TCP Short Connection time_wait problem solving method Daquan (3)--tcp_tw_recycle

TCP Short Connection time_wait problem solving method Daquan (4)--tcp_tw_reuse

"Experience Summary" failure caused by the tcp_tw_recycle parameter

Tcp_tw_recycle and tcp_timestamps cause connect failure issues

Tcp_tw_reuse, tcp_tw_recycle use scenes and precautions