Status and troubleshooting of TCP connections

Web Product performance Testing, there are many TCP connection problems, but also because of this problem, resulting in performance instability and so on, the data transmission between the client and server, and the transition between the connection state, which state is the normal state, which state is abnormal state, how to locate these problems, As well as the common tools, today a simple summary of these questions;

1

TCP Status Gain

1) Netstat-nat View the number of TCP states

2) Lsof-i:port can detect the condition of open sockets

3) Sar-n SOCK View the number of connections created by TCP

4) Tcpdump-iany TCP port 9000 for TCP ports 9000 for packet capture

2

TCP State Migration Roadmap

CLOSED: no connection status;

LISTENING: Listening for connection requests from a remote TCP port;

First, the server needs to open a socket for listening, the status is listen.

To provide a service to be in the listening state, the TCP state change is a port status changes, provide a service to open a port, for example: the WWW service is provided by default is 80 port, the FTP service provides the default port of 21, When the provided service is not connected, it is in the listening state. The FTP service is first in the listening (LISTENING) state after it is started. When listening to the listening state, the port is open, waiting for a connection, but not yet connected. Like the door of your house is open, but no one has come in.

Look at the listening state is the most important thing is to see which ports the machine opened, which programs are open, to close unnecessary ports is a very significant aspect of security, the service port corresponds to a service (application), stop the service to shut down the port, For example, to turn off Port 21, simply stop the FTP service in the IIS service. For information on this, see other articles.

If you unfortunately in the service Port Trojan, Trojan also open a port in the listening state.

Syn-sent: Client syn_sent Status:

Wait for a matching connection request after the connection request is sent: The client calls connect through the application for active open. The client TCP sends a SYN to request a connection. The status is then set to Syn_sent. Wait for a matching connection request after sending a connection request

When a connection is requested, the client first sends a synchronization signal to the machine to be accessed, at which time the state is syn_sent, and if the connection succeeds it becomes established, and under normal circumstances the syn_sent state is very short. For example, to visit the website http://www.baidu.com, if it is normal connection, the connection with TCPView to observe IE will find a quick change from syn_sent to established, indicating a successful connection. Syn_sent the state of the fast may not be seen.

If you find a lot of syn_sent appear, that generally have such a few cases, one is that you want to visit the site does not exist or bad line, the second is scanning software scan a network segment of the machine, there will be a lot of syn_sent, the other is probably the virus, for example, in the "Shock wave", When the virus attacks, it scans other machines, so many syn_sent will appear.

Syn-received: Server-side status SYN_RCVD

Wait for confirmation of the connection request after receiving and sending a connection request

When the server receives the synchronization signal sent by the client, the flag bit ACK and SYN 1 are sent to the client, and the server side is in the SYN_RCVD state, and if the connection succeeds, it becomes established, and under normal circumstances SYN_RCVD state is very short.

If you find that there are many SYN_RCVD states, your machine may be attacked by a SYN flood DOS (Denial of service attack).

Established: Represents an open connection.

The established state indicates that two machines are transmitting data, and the most important thing to see is which program is in the established state.

The server has many established states: Netstat-nat |grep 9502 or can be detected using lsof-i:9502.

Disconnect when the client is not actively close: the fin sent by the client is missing or not sent.

At this time if the client disconnects when the fin packet is sent, the server will be in the close_wait State;

At this time if the client disconnects when the fin packet is not sent, then the service side or display established status;

The resulting client reconnected to the server.

And the new connection on the client (that is, just broken back up) on the server is definitely established; If the client repeats this situation, there will be a large number of fake established connections and close_wait connections on the server side.

The end result is that the new client will not be able to connect, but using netstat still can see a connection has been established, and display established, but always cannot enter the program code

Fin-wait-1: Waiting for a remote TCP connection interrupt request, or confirmation of a previous connection interrupt request

The active close side application calls close, and its TCP sends a FIN request to actively close the connection and then enters the fin_wait1 state./* The socket is closed, and the connection is Shutting down. Pending connection interruption request for remote TCP, or confirmation of a previous connection interruption request */

If the server appears shutdown and then restarts, use Netstat-nat to view, you will see a lot of fin-wait-1 status. It is because the server currently has a lot of client connections, directly after shutting down the server, unable to receive the client's ACK.

Fin-wait-2: Waiting for connection interrupt request from remote TCP

After the active shutdown is received, an ACK is entered into the fin-wait-2, waiting for the connection to be interrupted from the remote TCP request

This is known as the semi-closed state, which is the state after the client and server two handshake when the connection is closed. In this state, the application also has the ability to accept data, but it has been unable to send data, but it is also possible that the client has been in a fin_wait_2 state, and the server has been in the Wait_close state, and until the application layer decides to close the state.

Close-wait: Waiting for a connection interrupt request from a local user

After the passive shutdown (passive close) side TCP receives FIN, an ACK is issued in response to the FIN request (its receive is also passed as a file terminator to the upper-level application) and enters Close_wait. Waiting for a connection interrupt request from a local user

CLOSING: Waiting for remote TCP to confirm connection interruption

Wait for the remote TCP acknowledgement of the connection interruption

Last-ack: Waiting for acknowledgement of the original connection interrupt request to the remote TCP

After a period of passive shutdown, the application that receives the file terminator will call close to close the connection. This causes its TCP to also send a FIN, which waits for the ACK of the other side. On the last-ack waiting for a connection interrupt request that was originally sent to remote TCP

When using the concurrency stress test, suddenly disconnecting the stress test client, the server will see a lot of last-ack.

Time-wait: Wait enough time to ensure that the remote TCP receives a connection interrupt request acknowledgement

After the active shutdown receives the FIN, TCP sends an ACK packet and enters the time-wait state, waiting for enough time to ensure that the remote TCP receives the acknowledgement of the connection interrupt request */

Time_wait wait state, this state is also called 2MSL State, said that after Time_wait2 sent the last ACK datagram, to enter the TIME_WAIT state, This state is prepared to prevent the last handshake from being delivered to the other party (note that this is not a four-time handshake, which is the fourth time that the handshake is insured). This state is to a large extent guaranteed that both sides can end normally, but the problem also comes. Due to the 2MSL status of the socket (socket is the meaning of IP and port pairs, socket), so that the application in 2MSL time is unable to use the same socket again, for the client program is good, but for the service program, such as httpd, it always use the same port to service, In 2MSL time, the boot httpd will be error (socket is used). To avoid this error, the server gives the concept of a quiet time, which means that in 2MSL time, although the server can be restarted, but the server will still wait for 2MSL time in the past before the next connection.

The state migration diagram of TCP is seemingly complex, but there are two lines in careful observation;

State migration diagram for client applications

The state of the client can be represented by the following process:

Closed->syn_sent->established->fin_wait_1->fin_wait_2->time_wait->closed

The above process is in the normal situation of the program should have a process, from the diagram in the book can be seen, when the connection is established, when the client receives the ACK of the SYN message, the client opens the data to connect interactively. The end-of-connection is usually the client's active end, and after the client finishes the application, it needs to go through the state of Fin_wait_1,fin_wait_2, which is the four-time handshake of the end connection mentioned earlier.

State migration diagram of the server

The status of the server can be represented by the following process:

Closed->listen->syn received->established->close_wait->last_ack->closed

When the connection is established, the server side enters the data interaction state after the third handshake, and the close connection is after the second handshake of the connection is closed (note that it is not the fourth time). After closing, it waits for the client to give the final ACK packet in order to enter the initial state.

Other State migrations

There are other state migrations, which are summarized for both the server and the client:

Listen->syn_sent, for this explanation is very simple, the server sometimes to open the connection.

Syn_sent->syn received, the server and the client in the Syn_sent state if received SYN datagram, you need to send the SYN ACK datagram and adjust its state to the SYN received state, ready to enter the established

Syn_sent->closed, in the case of a send timeout, is returned to the CLOSED state.

Syn_ receives->listen, if received by the RST packet, it returns to the LISTEN state.

Syn_ received->fin_wait_1, this migration is said, can not go to established state, and can jump directly to the fin_wait_1 state and wait to close.

3

TCP connection establishes three handshake

Client Connection server:

When the client side invokes the socket function call, the client side produces a socket in the closed state.

1) The first handshake: the client side calls the Connect function call, the system randomly assigns a port to the client, along with the parameters in the incoming connect (server's IP and port), which forms a connection four tuples, The client sends a TCP message with a SYN flag to the server. This is the message 1 in the three-time handshake process. The connect call allows the client side socket to be in the syn_sent state, waiting for the server to confirm; SYN: Synchronous sequence Number (Synchronize Sequence Numbers).

2) Second handshake: The server receives the SYN packet, must confirm the customer's SYN (ACK=J+1), simultaneously also sends a SYN packet (syn=k), namely the Syn+ack packet, at this time the server enters the SYN_RECV state;

3) The third handshake: the client receives the server's Syn+ack packet, sends the confirmation packet ack (ACK=K+1) to the server, the packet is sent, the client and the guest server enter the established state, and the handshake is completed three times. The connection is ready for read and write operations.

A full three-time handshake is also: request---answer---confirm again.

The TCP protocol completes the establishment of the connection through three message segments, which is called a three-time handshake (three-way handshake), as shown in the process.

The corresponding function interface:

Server:

When the server side invokes the socket function call, the server side produces a listening socket in the closed state

The server-side invokes the bind operation, associates the listening socket with the specified address and port, and then calls the Listen function, which assigns an incomplete queue and a completion queue, at which point the listening socket can accept the client connection and listen for the socket status in the listen state.

When the server side calls the accept operation, a completed client connection is taken out of the completion queue, and a session socket is generated in the server that is used to communicate with the client end socket, the state of which is establish.

As you can see, when the client calls connect, the connection request is triggered, the SYN J packet is sent to the server, then connect enters the blocking state, the server hears the connection request, receives the SYN J packet, calls the Accept function to receive the request to send SYN K to the client, ACK j+ 1, then accept into the blocking state, the client receives the server SYN K, after the ACK j+1, then connect returns, and the Syn K Confirmation, the server received an ACK k+1, accept return, this three times the handshake is completed, the connection is established.

We can see the specific process through the network capture package:

For example, our server opens 9502 ports. Use Tcpdump to grab the package:

Tcpdump-iany TCP Port 9502

Then we use Telnet 127.0.0.1 9502 to open the connection.:

Telnet 127.0.0.1 9502

14:12:45.104687 IP localhost.39870 > Localhost.9502:flags [S], seq 2927179378, Win 32792, options [MSS 16396,sackok,ts Val 255474104 ECR 0,nop,wscale 3], length 0 (1)

14:12:45.104701 IP localhost.9502 > Localhost.39870:flags [S.], seq 1721825043, Ack 2927179379, Win 32768, options [MS S 16396,sackok,ts Val 255474104 ECR 255474104,nop,wscale 3], length 0 (2)

14:12:45.104711 IP localhost.39870 > Localhost.9502:flags [.], ACK 1, Win 4099, options [Nop,nop,ts Val 255474104 ECR 255474104], length 0 (3)

14:13:01.415407 IP localhost.39870 > Localhost.9502:flags [P.], seq 1:8, ack 1, Win 4099, options [Nop,nop,ts Val 2554 78182 ECR 255474104], length 7

14:13:01.415432 IP localhost.9502 > Localhost.39870:flags [.], ACK 8, win 4096, options [Nop,nop,ts Val 255478182 ECR 255478182], length 0

14:13:01.415747 IP localhost.9502 > Localhost.39870:flags [P.], seq 1:19, ack 8, win 4096, options [Nop,nop,ts Val 255 478182 ECR 255478182], length 18

14:13:01.415757 IP localhost.39870 > Localhost.9502:flags [.], ACK, win 4097, options [Nop,nop,ts Val 255478182 ECR 255478182], length 0

We see (1) (2) (3) Three steps to establish TCP:

Handshake for the first time:

14:12:45.104687 IP localhost.39870 > Localhost.9502:flags [S], seq 2927179378

Client IP localhost.39870 (the port of the client is typically automatically assigned) sends a SYN packet (SYN=J) to the server localhost.9502 the server "

SYN's seq= 2927179378

Second handshake:

14:12:45.104701 IP localhost.9502 > Localhost.39870:flags [S.], seq 1721825043, Ack 2927179379,

The server receives the SYN packet, must confirm the customer's SYN (ACK=J+1), and also send itself a SYN packet (syn=k), that is, the Syn+ack package

SYN (ACK=J+1) =ack 2927179379 server host SYN packet (SYN=SEQ 1721825043)

Third handshake:

14:12:45.104711 IP localhost.39870 > Localhost.9502:flags [.], ACK 1,

The client receives the server's Syn+ack packet and sends an acknowledgment packet ack (ACK=K+1) to the server

After the client and server enter the established state, communication data can be interacted with. There is no relationship with the accept interface at this time, even if there is no accepte, there are 3 handshake completion.

Connection is not connected to the problem, usually network problems or network card overload or the number of connections is full.

Part of the purple background:

IP localhost.39870 > Localhost.9502:flags [P.], seq 1:8, ack 1, Win 4099, options [Nop,nop,ts Val 255478182 ECR 255474 104], Length 7

The client sends a 7-byte length of data to the server,

IP localhost.9502 > Localhost.39870:flags [.], ACK 8, win 4096, options [Nop,nop,ts Val 255478182 ECR 255478182], Leng Th 0

The server confirms to the customer that the data has been received

IP localhost.9502 > Localhost.39870:flags [P.], seq 1:19, ack 8, win 4096, options [Nop,nop,ts Val 255478182 ECR 25547 8182], Length 18

The server then writes data to the client at the same time.

IP localhost.39870 > Localhost.9502:flags [.], ACK, win 4097, options [Nop,nop,ts Val 255478182 ECR 255478182], Len Gth 0

The client confirms that the data has been received to the server

This is a reliable TCP connection, each communication needs to be confirmed by each other.

4

TCP connection terminated (four-time handshake)

Because TCP connections are full-duplex, each direction must be closed separately. The principle is that when a party completes its data sending task, it can send a fin to terminate the connection in this direction. Receiving a fin only means there is no data flow in this direction, and a TCP connection can still send data after receiving a fin. The first party to close will perform an active shutdown, while the other side performs a passive shutdown.

Establishing a connection requires three handshakes, while terminating a connection takes four handshakes, which is caused by a semi-shutdown of TCP (Half-close),

(1) Client A sends a fin to turn off customer A to Server B data Transfer (message segment 4).

(2) Server B receives this fin, which sends back an ACK, confirming that the serial number is received plus 1 (message Segment 5). As with Syn, a fin will occupy a sequence number.

(3) Server B closes the connection to client A and sends a fin to client a (message segment 6).

(4) Client A sends back ACK message confirmation, and sets the confirmation serial number to receive the serial number plus 1 (message segment 7).

corresponding function interface

The calling process is as follows:

• 1) When the client wants to close the connection between it and the server. The client (an application process) first calls close to actively close the connection, when TCP sends a FIN m;client end in the Fin_wait1 state.

• 2) When the server side receives the fin m, the passive shutdown is performed. The fin is confirmed and returned to the client ACK. When the server side returns to the client ACK, the client is in the FIN_WAIT2 state and the server is in the close_wait state. Its reception is also passed as a file terminator to the application process, because the receive of fin means that the application process can no longer receive additional data on the corresponding connection;

• 3) After a period of time, when the server side detects the client side of the shutdown operation (read returns to 0). A server-side call that receives a file terminator close closes its socket. This causes TCP on the server side to also send a fin N; At this point the server's status is Last_ack.

• 4) When the client receives fin from the server. The client-side socket is in the TIME_WAIT state, and it sends an ACK acknowledgement to the server side, which is in the closed state when the server side receives an ACK acknowledgement.

Uncle Ann said

Only to maintain the mentality of not willing to motivate themselves to continue to work forward, so always keep their own unwilling, to learn to let their progress, rather than fate and complaining

Status and troubleshooting of TCP connections