Kernel compilation resolves TCP connections large number of timewait

As the volume of business increased, the network pressure of the business Server increased, looking at the back-end server network connection status, found that the TIME_WAIT state connection is huge, time_wait occupies a large number of connection ports are not released, affecting the business service response speed. At the same time, a large number of each TCP connection has a data structure, called TCP Control block.time_wait when the data structure is not released. So when there are too many TCP connections, the memory may be occupied a lot.

Network-intensive background server, network connection status:

Av.com # av.com

Time_wait 47144 # time_wait 45920

Syn_sent 1 # syn_sent 1

Established 104 # established 94

LISTEN 7 # LISTEN 7

Av.com # av.com

Time_wait 45670 # time_wait 48396

Established 102 # established 99

LISTEN 7 # LISTEN 7

Av.com # av.com

Time_wait 45671 # time_wait 46268

Syn_sent 2 # established 100

Established # LISTEN 7

LISTEN 7

More than thirty thousand or forty thousand. At this time, we need to modify the Linux kernel tcp time wait, there is a can adjust the/etc/sysctl.conf network control related parameters, many online data.

However, the effect of modifying kernel parameters is minimal. Tested high concurrency server tuning kernel parameters, our side of the network environment, TCP time_wait can reduce about 10, the late business pressure is rising, the kernel parameter optimization of the server TCP time_wait will again crawl to a high point, such as.

How to solve the problem then? Let's take a look at what environment the time_wait will generate.

The process of TCP termination is as follows:

Server Client

--------------FIN--------------> Server:fin_wait_1

<-------------ACK---------------client:close_wait server:fin_wait_2

<-------------fin---------------client is closed after Fin is issued

--------------ack-------------> server enters time_wait state after issuing an ACK;

The default time of Time_wait is twice times the mls,mls (Maximum Segment Lifetime) is the longest time the TCP chip survives on the web, and its function is similar to the TTL of the IP packet. The primary function is to ensure that the closed TCP port is not immediately used. Because when there is a delay in the network, it is possible that when a port is shut down, there are some retransmission TCP slices in the network that are being sent to this port, and if the port immediately establishes a new TCP connection, it may be affected. So use twice times the MSL time to restrict the port from being used immediately.

Query the relevant data, a macro definition in the kernel, in the $KERNEL/include/net/tcp.h, this macro is the real control of TCP time_wait state time-out. The contents are as follows:

#define Tcp_timewait_len (60*HZ)

Modify this macro to define the numerical settings, according to our test, the commonly used values are the following three kinds: 30 seconds, 1 minutes, 2 minutes, can be measured according to the actual situation of the business, our side of the network pressure is large, through the final determination of pressure measurement set to 10 seconds, that is, the above changes to:

#define Tcp_timewait_len (10*HZ)

Then recompile the kernel, reboot the system to find the TIME_WAIT state of the short connection caused by the number of times the level of decline, we measured the effect as follows:

Kernel compilation Upgrade Reference:

http://michaelkang.blog.51cto.com/1553154/1266825

optimize system kernel Parameters host │ kernel compile optimized host

Av.com (hostname) │av.com (host name)

Time_wait 46366 │time_wait 9419

Established 99│FIN_WAIT1 3

LISTEN 7│established 118

av.com │listen 7

Time_wait 46125│av.com

Fin_wait1 1│time_wait 8950

Syn_sent 1│established 105

Established 99│listen 7

LISTEN 7│av.com

av.com │time_wait 9668

Time_wait 46591│established 114

Established 101│listen 7

LISTEN 7│av.com

av.com │time_wait 9224

Time_wait 44307│fin_wait1 1

Established 95│established 104

LISTEN 7│listen 7

av.com │av.com

Time_wait 46679│time_wait 9763

Established 99│FIN_WAIT1 2

LISTEN 7│established 116

av.com │listen 7

Time_wait 46833│av.com

Syn_sent 1│time_wait 9173

Established 99│FIN_WAIT1 1

LISTEN 7│established 103

av.com │syn_recv 1

Time_wait 45555│listen 7

Syn_sent 3│av.com

Established 104│time_wait 9598

LISTEN 7│established 117

av.com │listen 7

Time_wait 45214│av.com

Established 96│time_wait 9315

LISTEN 7│syn_sent 1

av.com │established 106

Time_wait 46498│listen 7

Syn_sent 1│av.com

Established 93│time_wait 9524

LISTEN 7│established 116

av.com │listen 7

Time_wait 46720│av.com

Syn_sent 1│time_wait 9110

Established 95│established 104

LISTEN 7│listen 7

Under the same network pressure situation, the optimized compiled kernel time_wait reduces nearly 5 times times, the effect is very significant.

This article is from the "Kang Jianhua" blog, make sure to keep this source http://michaelkang.blog.51cto.com/1553154/1683647

Kernel compilation Resolves a large number of TCP connections timewait