TCP Status explanation and troubleshooting

By understanding the various states of TCP, we can help to troubleshoot and locate network or system failures. (Summarize content on the network)

1. TCP Status

Linux View status command for TCP:1), Netstat-nat View the number of TCP states
2), Lsof-i:port can detect open socket status 3), Sar-n SOCK View TCP created Connections 4), Tcpdump-iany TCP port 9000 for TCP ports 9000 for the packet capture
LISTENING: Listens for connection requests from a remote TCP port.First, the server needs to open a socket for listening, the status is listen. To provide a service to be in the listening state, the TCP state change is a port status changes, provide a service to open a port, for example: the WWW service is provided by default is 80 port, the FTP service provides the default port of 21, When the provided service is not connected, it is in the listening state. The FTP service is first in the listening (LISTENING) state after it is started. When listening to the listening state, the port is open, waiting for a connection, but not yet connected. Like the door of your house is open, but no one has come in.
Look at the listening state is the most important thing is to see which ports the machine opened, which programs are open, to close unnecessary ports is a very significant aspect of security, the service port corresponds to a service (application), stop the service to shut down the port, For example, to turn off Port 21, simply stop the FTP service in the IIS service. For information on this, see other articles.
If you unfortunately in the service Port Trojan, Trojan also open a port in the listening state.
syn-sent: Client syn_sent Status:Wait for a matching connection request after the connection request is sent: The client calls connect through the application for active open. The client TCP sends a SYN to request a connection. The status is then set to Syn_sent. /*the socket is actively attempting to establish a connection. Wait for a matching connection request after sending a connection request */
When a connection is requested, the client first sends a synchronization signal to the machine to be accessed, at which time the state is syn_sent, and if the connection succeeds it becomes established, and under normal circumstances the syn_sent state is very short. For example, to access the Web site http://www.baidu.com, if it is normal connection, with TCPView observation IEXPLORE.EXE (IE) established connection will find a quick change from syn_sent to established, indicating a successful connection. Syn_sent the state of the fast may not be seen.
If you find a lot of syn_sent appear, that generally have such a few cases, one is that you want to visit the site does not exist or bad line, the second is scanning software scan a network segment of the machine, there will be a lot of syn_sent, the other is probably the virus, for example, in the "Shock wave", When the virus attacks, it scans other machines, so many syn_sent will appear.


Syn-received: Server-side status SYN_RCVD     Wait for confirmation of the connection request after receiving and sending a connection request   When the server receives the synchronization signal sent by the client, the flag bit ACK and SYN 1 are sent to the client, and the server side is in the SYN_RCVD state, and if the connection succeeds, it becomes established, and under normal circumstances SYN_RCVD state is very short.
If you find that there are many SYN_RCVD states, your machine may be attacked by a SYN flood DOS (Denial of service attack). The attack principle of SYN Flood is that the attack software sends a SYN connection request (the first step of the handshake) to the attacked server during the three handshake, but the address is forged, such as the attack software randomly forged 51.133.163.104, 65.158.99.152, and so on.     The server sends the flag ACK and SYN 1 to the client (the second step of the handshake) when it receives the connection request, but the IP addresses of these clients are bogus and the server cannot find the client at all, which means that the third step of the handshake cannot be completed. In this case the server side will typically retry (send Syn+ack to the client again) and wait for a period of time to discard the unfinished connection, the length of which we call Syn Timeout, which is generally the order of minutes (approximately 30 seconds-2 minutes) It is not a big problem for a user to have an exception that causes one of the server's threads to wait for 1 minutes, but if a malicious attacker simulates this situation in large numbers, the server will consume very much resources to maintain a very large semi-connected list----Tens of thousands of semi-connections, Even a simple save and traverse can consume a lot of CPU time and memory, not to mention the Syn+ack retry of the IP in this list. At this point, from a normal customer's point of view, the server is out of response, this situation we call doing:SYN flood attack on server side (SYN flood attack）


established: Represents an open connection. The established state indicates that two machines are transmitting data, and the most important thing to see is which program is in the established state.        The server has many established states: Netstat-nat |grep 9502 or can be detected using lsof-i:9502. Disconnect when the client is not actively close: the fin sent by the client is missing or not sent.

At this time if the client disconnects when the fin packet is sent, the server will be in the close_wait State;

At this time if the client disconnects when the fin packet is not sent, then the service side or display established status;

The resulting client reconnected to the server.

And the new connection on the client (that is, just broken back up) on the server is definitely established; If the client repeats this situation, there will be a large number of fake established connections and close_wait connections on the server side.

The end result is that the new client will not be able to connect, but with Netstat you can still see that a connection has been established and the established is displayed, but the program code is never entered.

fin-wait-1:Waiting for a remote TCP connection to interrupt the request, or an acknowledgement of a previous connection interruption request, the active shutdown (active close) side application calls Close, and its TCP sends a FIN request to actively close the connection before entering the FIN_WAIT1 state./* The socket is CLO SED, and the connection is shutting down. Waiting for a connection interrupt request from a remote TCP, or confirmation of a previous connection interrupt request * * If the server appears shutdown and then restarts, use Netstat-nat to view, you will see manyfin-wait-1The state. It is because the server currently has a lot of client connections, directly after shutting down the server, unable to receive the client's ACK.

fin-wait-2: Waiting for connection interrupt request from remote TCPAfter the active shut-off is received, the ACK is entered into the fin-wait-2./* Connection is closed, and the socket was waiting for a shutdown from the remote end. Waiting for a connection interrupt request from remote TCP */This is the state of the well-known semi-shutdown, which is the state after the client and server two-time handshake when the connection is closed. In this state, the application also has the ability to accept data, but it has been unable to send data, but it is also possible that the client has been in a fin_wait_2 state, and the server has been in the Wait_close state, and until the application layer decides to close the state.
close-wait: Waiting for a connection interrupt request from a local userAfter the passive shutdown (passive close) side TCP receives FIN, an ACK is issued in response to the FIN request (its receive is also passed as a file terminator to the upper-level application) and enters Close_wait. /* The remote end has a shut down and waiting for the socket to close. Waiting for a connection interrupt request from a local user */

CLOSING: Waiting for remote TCP to confirm connection interruptionRelatively uncommon./* Both sockets is shut down but we still don ' t has all of our data sent. Wait for the remote TCP acknowledgement of the connection interruption */

Last-ack: Waiting for acknowledgement of the original connection interrupt request to the remote TCPAfter a period of passive shutdown, the application that receives the file terminator will call close to close the connection. This causes its TCP to also send a FIN, waiting for the other person's ACK. It went into the last-ack. /* The remote end has a shut down, and the socket is closed. Waiting for acknowledgement. Wait for a connection interrupt request that was originally sent to remote TCP */When using the concurrency stress test, suddenly disconnecting the stress test client, the server will see a lot ofLast-ack.
time-wait: Wait enough time to ensure that the remote TCP receives a connection interrupt request acknowledgementAfter the active shut-off side receives fin, TCP sends an ACK packet and enters the time-wait state. /* The socket is waiting after close to handle packets still in the network. Wait enough time to ensure that remote TCP receives a connection interrupt request confirmation */TIM E_wait wait state, this state is called the 2MSL state, said that after Time_wait2 sent the last ACK datagram, to enter the TIME_WAIT state, this state is to prevent the last handshake of the datagram is not transmitted to the other side of the preparation (note this is not four times handshake , this is the fourth time that the handshake is insured). This state is to a large extent guaranteed that both sides can end normally, but the problem also comes.
Due to the 2MSL status of the socket (socket is the meaning of IP and port pairs, socket), so that the application in 2MSL time is unable to use the same socket again, for the client program is good, but for the service program, such as httpd, it always use the same port to service, In 2MSL time, the boot httpd will be error (socket is used).                     To avoid this error, the server gives the concept of a quiet time, which means that in 2MSL time, although the server can be restarted, but the server will still wait for 2MSL time in the past before the next connection. For details, please see:Time_wait caused cannot assign requested address error

CLOSED: No connection StatusAfter receiving the ACK packet, the passive shut-off end enters the state of the closed. The connection ends./* The socket is not being used. No connection Status */

2. TCP State Migration Roadmap

Client/server Two routes describe the TCP state Migration roadmap:

This is a seemingly complex state migration diagram, because it contains two parts---the state migration of the server and the State migration of the client, if from a certain point of view this figure, it will be clear that the server and the client is not absolute, the data is sent to the client, The server is the one that accepts the data.State Migration diagram for client applicationsThe state of the client can be represented by the following process: closed->syn_sent->established->fin_wait_1->fin_wait_2->time_wait->closed The above process is in the normal situation of the program should have a process, from the diagram in the book can be seen, when the connection is established, when the client receives the ACK of the SYN message, the client opens the data to connect interactively. The end-of-connection is usually the client's active end, and after the client finishes the application, it needs to go through the state of Fin_wait_1,fin_wait_2, which is the four-time handshake of the end connection mentioned earlier.
State Migration diagram of the serverThe status of the server can be represented by the following process: Closed->listen->syn received->established->close_wait->last_ack->closed in the establishment When connecting, the server side enters the data interaction state after the third handshake, while closing the connection is after the second handshake of the connection is closed (note that it is not the fourth time). After closing, it waits for the client to give the final ACK packet in order to enter the initial state.
Other State migrationsThere are other state migrations, these state migrations for the server and the client are summarized as follows Listen->syn_sent, for this explanation is very simple, the server sometimes to open the connection. Syn_sent->syn received, the server and the client in the Syn_sent state if received SYN datagram, you need to send the SYN ACK datagram and adjust its state to the SYN received state, ready to enter the established syn_sent- >closed, in the case of a send timeout, is returned to the CLOSED state. Syn_ receives->listen, if received by the RST packet, it returns to the LISTEN state. Syn_ received->fin_wait_1, this migration is said, can not go to established state, and can jump directly to the fin_wait_1 state and wait to close.




How to engrave this picture firmly in the mind. Then you must have a deep understanding of each state of the graph and the process of conversion, and not just stay in the smattering. Below is a detailed analysis of the 11 states of this graph in order to enhance memory. But before that, let's review the three handshake process for TCP to establish a connection, and the four-time handshake process for closing the connection.

3, TCP connection to establish three handshake TCP is a connection-oriented protocol, so before connecting the two parties to send data, you need to first establish a connection. Client Connection server:

When the client side invokes the socket function call, the client side produces a socket in the closed state.
(1) The first handshake: the client side also calls the Connect function call, the system for the client randomly assigned a port, along with the parameters in the incoming connect (server IP and Port), which formed a connection four tuples, The client sends a TCP message with a SYN flag to the server. This is the message 1 in the three-time handshake process. The connect call allows the client side socket to be in the syn_sent state, waiting for the server to confirm; SYN: Synchronous sequence Number (Synchronize Sequence Numbers).

(2) Second Handshake : The server receives the SYN packet, must confirm the customer's SYN (ACK=J+1), simultaneously also sends a SYN packet (syn=k), namely the Syn+ack packet, at this time the server enters the syn_recv state;

(3) The third handshake: the client receives the server's Syn+ack packet, sends the confirmation packet ack (ACK=K+1) to the server, the packet is sent, the client and the guest server enter the established state, and the handshake is completed three times. The connection is ready for read and write operations.

A full three-time handshake is also: request---answer---confirm again.

The TCP protocol completes the establishment of the connection through three message segments, which is called a three-time handshake (three-way handshake), as shown in the following figure. The corresponding function interface:

2) Server
When the server side invokes the socket function call, the server side produces a listening socket in the closed state
The server-side invokes the bind operation, associates the listening socket with the specified address and port, and then calls the Listen function, which assigns an incomplete queue and a completion queue, at which point the listening socket can accept the client connection and listen for the socket status in the listen state.
When the server side calls the accept operation, a completed client connection is taken out of the completion queue, and a session socket is generated in the server that is used to communicate with the client end socket, the state of which is establish. As can be seen from the figure, when the client calls connect, the connection request is triggered, the SYN J packet is sent to the server, and connect enters the blocking state, and the server hears the connection request, receives the SYN J packet, calls the Accept function to receive the request to send SYN K to the client, ACK j+ 1, then accept into the blocking state, the client receives the server SYN K, after the ACK j+1, then connect returns, and the Syn K Confirmation, the server received an ACK k+1, accept return, this three times the handshake is completed, the connection is established.
We can see the specific process through the network capture package: For example, our server opens 9502 port. Use Tcpdump to grab the package:tcpdump-iany TCP port 9502Then we use Telnet 127.0.0.1 9502 to open the connection.: Telnet 127.0.0.1 9502 14:12:45.104687 IP localhost.39870 > Localhost.9502:flags [S], Seq 2927179378, Win 32792, options [MSS 16396,sackok,ts val 255474104 ECR 0,nop,wscale 3], length 0(1)
14:12:45.104701 IP localhost.9502 > Localhost.39870:flags [S.], seq 1721825043, Ack 2927179379, Win 32768, options [MS S 16396,sackok,ts Val 255474104 ECR 255474104,nop,wscale 3], length 0(2)
14:12:45.104711 IP localhost.39870 > Localhost.9502:flags [.], ACK 1, Win 4099, options [Nop,nop,ts Val 255474104 ECR 255474104], length 0(3)
14:13:01.415407 IP localhost.39870 > Localhost.9502:flags [P.], seq 1:8, ack 1, Win 4099, options [Nop,nop,ts Val 2554 78182 ECR 255474104], length 7
14:13:01.415432 IP localhost.9502 > Localh