Linux system receives SYN but does not return syn+ack problem troubleshooting

One, background:

This afternoon found that a machine on the line from the office network is not logged on and all TCP ports are Telnet, but through the same machine room other machines can normally access to the problem of the machine. So immediately in this problem server-side packet analysis, found that the problem is as follows:
The server side received a SYN packet from the local PC, but did not return the Syn+ack packet, so it is a server-side system issue. Tcpdump grab bag as follows:

650) this.width=650; "Src=" https://s5.51cto.com/wyfs02/M01/9E/00/wKioL1mJym_yXf-IAAZzy5AQiS4374.jpg-wh_500x0-wm_ 3-wmp_4-s_975298027.jpg "title=" 1.jpg "alt=" Wkiol1mjym_yxf-iaazzy5aqis4374.jpg-wh_50 "/>

Second, troubleshoot

1, found that the system does not have any load

2, the network card has not lost the packet

3,iptables strategy is fine, too.

4, the System SYN_RECV connection is very few, also no overrun

5, the System file descriptor and other resources are also no problem

No prompts or error messages in 6,messages and DMESG

Third, through Google to help

found that the same person met the problem:

is by adjusting sysctl-w net.ipv4.tcp_timestamps=0 or sysctl-w net.ipv4.tcp_tw_recycle=0 to solve this problem, so I continue to investigate.

In the process of querying these two parameters, the cause of the problem is as follows:

Discovery is a problem caused by the Linux tcp_tw_recycle/tcp_timestamps setup. Because in the Linux kernel source found tcp_tw_recycle/tcp_timestamps are open conditions, 60s within the same source IP host socket connect request in the timestamp must be incremented. After testing, my side CENTOS6 system (kernel 2.6.32) and CENTOS7 system (kernel 3.10.0) have this problem.

     source function: Kernel 2.6.32 tcp_v4_conn_request (), the function is the TCP layer three-time Handshake SYN Packet processing function (server side);      Source Fragment:       if  (tmp_opt.saw_tstamp &&             tcp_death_row.sysctl_tw_recycle  &&             (dst = inet_csk_ Route_req (sk, req))  != NULL &&              (Peer = rt_get_peer (struct rtable *) DST)  != NULL  &&            peer->v4daddr == &NBSP;SADDR)  {            if  (get_ Seconds ()  < peer->tcp_ts_stamp + TCP_PAWS_MSL &&                  (S32) (peer->tcp_ts - req->ts_recent)  >                              tcp_paws_window)  {      &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;NET_INC_STATS_BH (Sock_net (SK),  LINUX_ mib_pawspassiverejected);                 goto drop_and_release;             }        }                 tmp_opt.saw_tstamp: the socket supports tcp_timestamp         sysctl_tw_recycle: tcp_tw_recycle option for native system open          tcp_paws_msl:60s, the condition determines that theThe last TCP communication for the source IP occurred within 60s         tcp_paws_window:1, This condition determines that the last TCP traffic for the source IP timestamp  is greater than   this TCP

Summarize:

My side and other colleagues access the problem server through the company exit (NAT gateway only 1 IP addresses), because timestamp time for the system to boot to the current time, so I and other colleagues timestamp certainly not the same, according to the above SYN packet processing source, in Tcp_tw_ Recycle and tcp_timestamps simultaneously open conditions, timestamp large host access Servern successful, while Timestmap small host access failed. And, I found two machines in the office network to reproduce the problem 100%.

Solve:

# echo "0" >/proc/sys/net/ipv4/tcp_tw_recycle

Four, expand

1,net.ipv4.tcp_timestamps

The essence of Tcp_timestamps is to record the sending time of a packet. The basic steps are as follows:

1. When sending data, the sender puts a timestamp (indicating the sending time) in the package

2. After receiving the packet, the receiver returns the received timestamp to the sender in the corresponding ACK packet (echo back)

3. After sending an ACK packet, an accurate RTT can be obtained with the timestamp in the Now-ack packet at the current moment.

Of course, the fluctuation of RTT should be considered in practical application, so there is a follow-up (round-trip time measurement) Rttm mechanism.

TCP timestamps Option (TSopt) is specifically designed as follows

kind: 8              //  tag unique option type, such as Window scale is 3length: 10 bytes    //   mark the number of bytes of the timestamps option +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++| kind=8 |  Length=10 | TS Value  (TSval)  | TS ECho Reply  (TSECR)  | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++    1           1              4                        4 

Timestamps a two-way option, when a party does not open, both sides will deactivate the timestamps. For example, the client sends a SYN packet with the timestamp option, but the server side does not have this option turned on. The syn-ack of the reply will not have the timestamp option, and the ACK of the client's subsequent reply will not have the timestamp option. Of course, if the client sends a SYN packet without timestamp, the timestamp will be deactivated in both directions.

The value of timestamps in the TCP packet is the (millisecond) timestamp of the system boot time to the present time.

Parameters:

0: Deactivate

1: Enable

2,net.ipv4.tcp_tw_recycle

TCP connections in time_wait that are specified in the TCP specification must wait 2MSL time. However, in Linux, if the tcp_tw_recycle,time_wait TCP connection is turned on, it will not wait for 2MSL time (instead of RTO or 60s) to achieve the purpose of quickly reusing (reclaiming) a TCP connection in the TIME_WAIT state. This can cause the connection to receive data that was previously connected. To do this, Linux will record the peer information of the TIME_WAIT connection, including the IP address, timestamp, and so on, in the case of opening tcp_tw_recycle. this way, when the kernel receives a SYN packet of the same IP, it compares the timestamp, checks whether the SYN packet's timestamp is lagging, and discards it if it lags (considered the data of the old connection). This is not a problem in most cases, but for our actual client-server service, the users who access our services are generally located after Nat, and if there are multiple users accessing the same service after NAT, it is possible that a connection with a timestamp lag is discarded.

Parameters:

0: Deactivate

1: Enable

Reference:

Https://serverfault.com/questions/235965/why-would-a-server-not-send-a-syn-ack-packet-in-response-to-a-syn-packet

http://hustcat.github.io/tcp_tw_recycle-and-tcp_timestamp/

This article is from the "Good" blog, please be sure to keep this source http://leejia.blog.51cto.com/4356849/1954628

Linux system receives SYN but does not return syn+ack problem troubleshooting