The processing of CentOS mass time_wait

Today, it is found that the Linux machine cannot be connected with SSH and can not ping the machine, behind by NETSTAT-ANP | grep time_wait | Wc-l commands to see the number of time_wait connections, let's take a look at a lot of time_wait problem solving.

word-spacing:0px; -webkit-text-stroke-width:0px; -webkit-text-size-adjust:auto "> Phenomenon:

1, the external machine can not properly connect SSH

2, inside outside can not normal ping pass, domain name also cannot normal parse.

Troubleshoot the problem:

by NETSTAT-ANP | grep time_wait | Wc-l command to see the number, found that the number of connections time_wait more than 18000 is too exaggerated.

1, the initial suspicion is that the program did not close the connection, Codereview two times, found that has been normally closed.

2, online to see the cause of time_wait, it may be because the server actively shut down the connection caused by time_wait generation.

3. Find Time_wait Solution:

Pse:separate; Font-family:tahoma; White-space:normal; Orphans:2; Letter-spacing:normal; Font-weight:normal; word-spacing:0px; -webkit-text-stroke-width:0px; -webkit-text-size-adjust:auto; -webkit-text-decorations-in-effect:none "> Discovery System has a large number of time_wait state connections, by adjusting the kernel parameters to resolve,
Vi/etc/sysctl.conf

Edit the file and add the following:
Net.ipv4.tcp_syncookies = 1
Net.ipv4.tcp_tw_reuse = 1
Net.ipv4.tcp_tw_recycle = 1
Net.ipv4.tcp_fin_timeout = 30

Then execute/sbin/sysctl-p to let the parameters take effect.

After the configuration, the temporary problem is resolved, and then see the number of time_wait quickly decline.

Key commands:

1, Netstat-n | awk '/^tcp/{++state[$NF]} END {for (key in) print key, "T", State[key]} '

You will get a result similar to the following, the exact number will be different:

Last_ack 1
Syn_recv 14
Established 79
Fin_wait1 28
Fin_wait2 3
CLOSING 5
Time_wait 1669

Status: Description
CLOSED: No connection is active or in progress
LISTEN: The server is waiting to enter the call
SYN_RECV: A connection request has arrived, waiting for confirmation
Syn_sent: Application has started, open a connection
Established: Normal data transfer status
FIN_WAIT1: Application says it's done
Fin_wait2: The other side has agreed to release
Itmed_wait: Waiting for all packets to die
CLOSING: Both sides try to close simultaneously
Time_wait: The other side has initialized a release
Last_ack: Waiting for all packets to die

2, Sysctl-a | grep Time | grep wait
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
Net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120

Field Description:Net.ipv4.tcp_syncookies = 1 means that Syn Cookies are turned on. When a SYN wait queue overflow occurs, cookies are enabled to protect against a small number of SYN attacks, the default is 0, which means close; Net.ipv4.tcp_tw_reuse = 1 means turn on reuse. 　　Allow time-wait sockets to be reused for new TCP connections, which defaults to 0, which means shutdown, net.ipv4.tcp_tw_recycle = 1 for fast recycling of time-wait sockets on TCP connections, and 0 for shutdown by default. 　　Net.ipv4.tcp_fin_timeout = 30 means that if the socket is closed by the local side, this parameter determines how long it remains in the fin-wait-2 state. Net.ipv4.tcp_keepalive_time = 1200 indicates the frequency at which TCP sends keepalive messages when KeepAlive is employed. 　　The default is 2 hours, which is changed to 20 minutes. Net.ipv4.ip_local_port_range = 1024 65000 indicates the range of ports used for an outward connection. 　　Small by default: 32768 to 61000, 1024 to 65000.  　　Net.ipv4.tcp_max_syn_backlog = 8192 Indicates the length of the SYN queue, the default is 1024, and the queue length is 8192, which can accommodate more network connections waiting to be connected. Net.ipv4.tcp_max_tw_buckets = 5000 indicates that the system maintains the maximum number of time_wait sockets at the same time, and if this number is exceeded, the time_wait socket is immediately cleared and a warning message is printed. The default is 180000, which changes to 5000. For Apache, Nginx and other servers, the parameters of the last few lines can be a good way to reduce the number of time_wait sockets, but for squid, the effect is not small. This parameter controls the maximum number of time_wait sockets, preventing squid servers from being dragged to death by a large number of time_wait sockets. Data just printed: time_wait 80
Fin_wait1 9
Established 317
FIN_WAIT2 710: determined by Net.ipv4.tcp_fin_timeout = 30, at this time to maintain 30 seconds, calculated, there are 130 http links, 5 seconds to refresh, 130* (30/5) = 780, basic compliance.
SYN_RECV 2
CLOSING 1
Last_ack 1