Troubleshoot excessive number of TCP connections

Troubleshoot excessive number of TCP connections

TCP state Migration, Close_wait & Fin_wait2 Issues

TCP State Migration

Everyone to netstat-a command very familiar, but, you have not noticed the state of a column, basically shows the established,time_wait,close_wait, and so on, these exactly what meaning, in this article, I will elaborate.

It is very clear that the TCP initialization connection three times handshake: send the SYN packet, then return the Syn/ack packet, then send the ACK packet, the connection is formally established. But there's a bit of a discrepancy here, and when the requestor receives the Sys/ack packet, it starts to establish the connection, and the requestor's third handshake ends before the connection is established. But do you understand how closing a connection works? Close the connection to four handshake: send fin, ACK, FIN, ACK, four handshake!! Why, because the TCP connection is full-duplex, I close your connection, not equal to the connection you closed.

Client TCP State Migration:

Closed->syn_sent->established->fin_wait_1->fin_wait_2->time_wait->closed

Server TCP State Migration:

Closed->listen->syn received->established->close_wait->last_ack->closed

When the client starts to connect, the server is still in listening,

After the client sends a SYN packet, he is in the Syn_sent state, the server is in SYS received State,

Then confirm each other into the connection state established.

When the client requests to close the connection, the client sends a FIN packet, the client enters the fin_wait_1 state, waits for the other party's confirmation packet,

The server sends an ACK packet to the client, the client receives an ACK packet after the end of the Fin_wait_1 state, enters the fin_wait_2 state, waits for the server to send a shutdown request,

After the server sends a FIN packet, it enters the close_wait state,

When the client receives the fin packet from the server, the Fin_wait_2 state ends, and the server-side fin packet is given a confirmation packet, and the client enters Time_wait,

When the server receives the confirmation packet, the close_wait state is ended,

At this time the server side really shut down the connection. But the client is still in the TIME_WAIT state,

When is it over? I'm here to talk about another term: 2MSL waiting state, in fact, Time_wait is 2MSL waiting state,

Why set this state, because there is enough time for the ACK packet to reach the server side, if the server side is confiscated to the ACK packet, timed out, and then resend a fin packet until the server receives an ACK packet.

The time_wait status wait time is twice times less than any requests that are connected after a TCP restart.

We have not found a problem: if the other side in the third handshake, such as the issue of fin bag, do not know what reason to lose this package, but this side has been in the Fin_wait_2 state, and TCP/IP does not set the expiration time of this state, he will keep this state, More and more fin_wait_2 states can cause the system to crash.

The problem I encountered above is that the TCP end process has not been completed, causing the connection not to be released. The client is now actively disconnected, the process is as follows

As shown,

Client message Server

Close ()

------FIN------->

Fin_wait1 close_wait

<-----ACK-------

Fin_wait2

Close ()

<------FIN------

Time_wait Last_ack

------ACK------->

CLOSED

CLOSED

Since the server's socket has not been called off while the client is closed,

Causes the server-side connection to be in a "pending" state, while the client is waiting for a reply.

Typical characteristics of this problem are:

One end is in Fin_wait2 and the other end is in close_wait.

However, the fundamental problem is that the procedure is not well written and needs to be improved

-------------------------------------------------------------------------

Close_wait,tcp's cancer, a friend of TCP.

Reason for close_wait status

First we know that if our server program Apache is in the close_wait state, it means that the socket is passively closed!

Because if the client side actively disconnects the current connection, then the two sides to close this TCP connection requires a total of four packet:

Client---> FIN---> Server

Client <---ACK <---Server

The client side is in the fin_wait_2 state while the server program is in the Close_wait state.

Client <---FIN <---Server

When the Server sends fin to client,server it is set to Last_ack state.

Client---> ACK---> Server

When the client responds with an ACK, the server's socket is actually set to the closed state.

The Server program is in the close_wait state instead of the Last_ack state, stating that the fin has not been sent to the client, so there may be a lot of data to be sent or other things to do before closing the connection, resulting in the absence of this fin packet.

Usually, a close_wait will last for at least 2 hours. If a rogue specifically wrote a program that gives you a bunch of close_wait and consumes your resources, then it's usually not until the release that the system has solved the crash.

This time can only be shortened by modifying the parameters of TCP/IP: Modifying the tcp_keepalive_* series parameters helps to solve this problem.

The solution to this problem is to modify the system parameters, the system default timeout time is 7,200 seconds, that is, 2 hours, this is too large, you can modify the following parameters:

Sysctl-w net.ipv4.tcp_keepalive_time=30

Sysctl-w net.ipv4.tcp_keepalive_probes=2

Sysctl-w net.ipv4.tcp_keepalive_intvl=2

Then, execute the sysctl command to make the changes effective.

The connection process is represented by a series of States, which are:

Listen,syn-sent,syn-received,established,fin-wait-1,fin-wait-2,close-wait,closing,last-ack,time-wait and closed.

The meanings of each state are as follows:

LISTEN-listens for connection requests from remote TCP ports;

Syn-sent-waits for a matching connection request after the connection request is sent;

Syn-received-Waits for confirmation of the connection request after receiving and sending a connection request;

established-represents an open connection, the data can be transmitted to the user;

Fin-wait-1-a connection interruption request waiting for a remote TCP, or confirmation of a previous connection interruption request;

Fin-wait-2-Waits for a connection interrupt request from a remote TCP;

Close-wait-waits for a connection interruption request from a local user;

CLOSING-Wait for the remote TCP acknowledgement of the connection interruption;

Last-ack-Acknowledgement of the connection interrupt request that was originally sent to the remote TCP;

Time-wait-wait enough time to ensure that the remote TCP receives the acknowledgement of the connection interrupt request;

CLOSED-no connection status;

The TCP connection process is the transition of the state, prompting the user to invoke the state transition:

Open,send,receive,close,abort and status;

Data segments transmitted over, particularly those including the following tagged data segments syn,ack,rst and fin;

There is a timeout, and the TCP status changes as mentioned above.

Status transition diagram for TCP connections

This figure n many people know that it helps to troubleshoot and locate networks or systems, but how do you keep this picture in mind? Then you must have a deep understanding of each state of the graph, and the process of conversion, and not just stay in the smattering. Here is a detailed explanation of the 11 states of this picture in order to enhance the memory! But before that, let's review the three handshake process for TCP to establish a connection, and the four-time handshake process for closing the connection.

1. Establish Connection Agreement (three handshake)

(1) The client sends a TCP message with a SYN flag to the server. This is the message 1 in the three-time handshake process.

(2) server-side response to the client, this is the 2nd message in the three handshake, the message with both an ACK flag and a SYN flag. So it represents the response to the client's SYN message, and it also flags the SYN to the client and asks the client if it is ready for data communication.

(3) The customer must again respond to the service segment an ACK message, which is the message segment 3.

2. Connection termination protocol (four handshake)

Because TCP connections are full-duplex, each direction must be closed separately. The principle is that when a party completes its data sending task, it can send a fin to terminate the connection in this direction. Receiving a fin only means there is no data flow in this direction, and a TCP connection can still send data after receiving a fin. The first party to close will perform an active shutdown, while the other side performs a passive shutdown.

(1) The TCP client sends a fin to shut down the client-to-server data transfer (message segment 4).

(2) The server receives this fin, it sends back an ACK, confirms that the serial number is the received sequence number plus 1 (message Segment 5). As with Syn, a fin will occupy a sequence number.

(3) The server shuts down the client connection and sends a FIN to the client (message segment 6).

(4) The customer segment sends back ACK message confirmation, and the confirmation serial number is set to receive the serial number plus 1 (message segment 7).

CLOSED: This is nothing to say, it means the initial state.

LISTEN: This is also very easy to understand a state, that the server side of a socket is listening state, can accept the connection.

SYN_RCVD: This status means that the SYN message is received, under normal circumstances, this state is the server side of the socket in the establishment of a TCP connection during the three handshake session process of an intermediate state, very short, basically with netstat you are very difficult to see this state, Unless you specifically write a client-side test program, deliberately not send the last ACK message during the three TCP handshake. Therefore, when the ACK message is received from the client, it goes into the established state.

Syn_sent: This state is echoed with the syn_rcvd thinking back, when the client socket performs a connect connection, it sends the SYN message first, so it then enters the syn_sent state and waits for the server to send the 2nd message in the three-time handshake. The Syn_sent status indicates that the client has sent a SYN message.

Established: This is easy to understand, indicating that the connection has been established.

Fin_wait_1: This state should be well explained, in fact, the real meaning of fin_wait_1 and fin_wait_2 state is to wait for each other's fin message. The difference between the two states is: The fin_wait_1 state is actually when the socket in the established state, it would like to actively close the connection, send a FIN message to the other side, when the socket is entered into the fin_wait_1 state. And when the other party responds to the ACK message, then into the fin_wait_2 state, of course, under the actual normal circumstances, regardless of the circumstances of each other, should immediately respond to the ACK message, so fin_wait_1 state is generally more difficult to see, and Fin_wait_ 2 states can also sometimes be seen with netstat.

Fin_wait_2: Above has explained in detail this state, actually fin_wait_2 the socket in the state, indicates the half connection, also namely has the party request close connection, but also tells the other side, I temporarily also some data need to transmit to you, later closes the connection again.

Time_wait: Said to receive the other side of the fin message, and sent out an ACK message, just wait for 2MSL to return to the closed usable state. If the fin_wait_1 state, received the other side with the FIN flag and the ACK flag message, you can directly into the time_wait state, without having to go through the fin_wait_2 state.

CLOSING: This kind of state is special, the actual situation should be very rare, belong to a relatively rare exception state. Normally, when you send a fin message, it is supposed to receive (or receive) the other's ACK message before receiving the other's fin message. But closing status indicates that you send fin message, and did not receive the other's ACK message, but also received the other side of the fin message. Under what circumstances will this happen? In fact, it is not difficult to come to a conclusion: that is, if the two sides close a socket at the same time, then there is a situation where both sides send the fin message, there will be a closing state, indicating that both sides are shutting down the socket connection.

Close_wait: The meaning of this state is actually expressed in waiting to be closed. How do you understand it? When the other side close a socket to send fin message to yourself, you will undoubtedly respond to an ACK message to each other, then enter into the close_wait state. Next, the real thing you really need to consider is whether you still have the data sent to the other person, if not, then you can close the socket, send fin messages to each other, that is, close the connection. So what you need to accomplish in the close_wait state is waiting for you to close the connection.

Last_ack: This state is still relatively easy to understand, it is the passive close side after sending fin messages, and finally wait for each other's ACK message. When an ACK message is received, it is also possible to enter the closed available state.

Finally there are 2 questions answered, my own analysis of the conclusion (not necessarily guaranteed 100% correct)

1, why to establish a connection agreement is three handshake, and close the connection is four handshake it?

This is because the socket in the listen state of the server is sent in a message after it receives a request for the connection of the SYN message, and it can put the ACK and SYN (ACK response, and SYN synchronous). However, when the connection is closed, when receiving the other's fin message notification, it simply means that no data is sent to you, but not all of your data are sent to the other side, so you may not immediately close the socket, that is, you may also need to send some data to the other side, Send the FIN message to the other side to show that you agree that you can now close the connection, so it is here that the ACK message and fin messages are sent separately in most cases.

2, why time_wait state also need to wait 2MSL to return to closed status?

This is because: although both sides agree to close the connection, and the handshake of the 4 messages are also coordinated and sent, can be directly back to the closed state (like from the Syn_send state to establish state); but because we have to assume that the network is unreliable, You cannot guarantee that the last ACK message you send will be received by the other side, so the other side in the Last_ack state of the socket may be due to the timeout did not receive an ACK message, and the re-send fin message, so this time_wait state function is used to resend the possible missing ACK message, And be assured of this.

When disconnecting, when initiating the left side of the active shutdown, this side sends a fin past,

This side of the right passive shutdown responds with an ACK, which is the TCP response, not the application sent,

At this point, the passive closed side is in the close_wait state.

If this side of the passive shutdown no longer calls Closesocket, then he will not send the next fin, causing himself to always be close_wait.

Only the passive shutdown of this party called the Closesocket, will send a fin to the active shutdown of this side, but also make their state changes to Last_ack.

For example, the passive shutdown is the client.

When the caller calls Closesocket, your program is

int nret = recv (s,....);

if (nret = = socket_error)

{

Closesocket (s);

return FALSE;

}

A lot of people just forget the phrase closesocket, this code is too common.

My understanding,

When the active shutdown side sends the fin to the passive close side, the passive off side of TCP immediately responds with an ACK past, and submits an error to the application above,

Causes the send or recv of the socket above to return SOCKET_ERROR.

Normally, if Closesocket is called after returning Socket_error, then TCP on the passive-closed side sends a fin past, and its state changes to Last_ack.

A large number of close_wait examples and solutions are found on the server (examples from the internet, almost)

[Email protected]] $/usr/sbin/lsof-i | grep 6800

Oracle 22725 oracle9i 3u IPv4 18621468 TCP rhel3:6800 (LISTEN)

Oracle 22725 oracle9i 4u IPv4 18621469 TCP rhel3:6800->rhel3:2174 (close_wait)

Oracle 22725 oracle9i 8u IPv4 18621568 TCP rhel3:6800->rhel3:2175 (close_wait)

Oracle 22725 oracle9i 9u IPv4 18621578 TCP rhel3:6800->rhel3:2176 (close_wait)

Oracle 22726 oracle9i 3u IPv4 18621468 TCP rhel3:6800 (LISTEN)

Oracle 22726 oracle9i 4u IPv4 18621469 TCP rhel3:6800->rhel3:2174 (close_wait)

Oracle 22726 oracle9i 8u IPv4 18621568 TCP rhel3:6800->rhel3:2175 (close_wait)

Oracle 22726 oracle9i 9u IPv4 18621578 TCP rhel3:6800->rhel3:2176 (close_wait)

[Email protected] oracle9i]$ kill-9 22725

# 22725, 22726 is the process number (PID) that uses the 6800 port.

[Email protected] oracle9i]$/usr/sbin/lsof-i | grep 6800

When the process is killed, all the link handles that are occupied are released.

The cause of the problem is everywhere on the web, that is, the client side of the socket is not close on the exit.

The above sentence is not very accurate, it should be passive close the connection at one end without closesocket on the exit, when the passive closed one end is in the close_wait state

Troubleshoot excessive number of TCP connections