TCP and HTTP KeepAlive

I've had a problem with my recent job and wanted to write it down, and the scenario is this:

From the above figure can be seen, the user through Client access is LVS VIP, VIP back-end Mount Realserver is Nginx server. Client can be a browser or a client program. In general, this architecture will not cause problems, but if the client side of the request to send the Nginx,nginx back end will take some time to return the results, more than 1 minutes and 30 seconds there is a problem, using LVS as a load balancing device to see the phenomenon is 1 minutes 30 seconds later, The client and Nginx links are disconnected and no data is returned. The reason is LVS default to keep TCP session 90s, more than 90s no TCP message transmission on the link, LVS will send reset message to both ends of the link. The reason for LVs to believe that we all know twos, I know the reasons are mainly two points:

1. Save load Balancing equipment resources, each TCP/UDP link will create a session structure on the load balancing device, if the link is kept open, this session structure information will eventually consume all resources, so it must be released. 2. Another release can protect the back-end resources, if the attacker through an empty link, link to nginx, if Nginx did not do appropriate protection, Nginx will be unable to provide services because of too many links.

This problem is not only on the LVS, before the commercial load balancing equipment F5 encountered the same problem, F5 session disconnected way and LVs a little different, F5 will not actively send reset to the ends of the link, the session disappeared, When a link to send a message again will receive F5 reset, after the phenomenon is sent again the end of the message TCP link state has been disconnected, and the other end is still establish state.

Once you know the reason for the load balancing device, the first reaction is to solve it by opening the keepalive. This should be the end of the problem, but I found that some time there is always someone to mention the problem of keepalive, and even found that due to KeepAlive's understanding is not correct to waste a lot of resources, could have used the application of LVS in the public net subsidence area, or replaced by commercial F5 equipment ( The session disconnect time for the F5 device is a bit longer and should be 5 minutes by default. So I decided to write a blog about the keepalive knowledge I know.

Why should there be keepalive.

Before we talk about KeepAlive, we should first understand the simple TCP knowledge (the knowledge is very simple, the master directly ignores). The first thing to be clear is that there is no "request" on the TCP layer, and it is wrong to hear that a request is sent at the TCP layer often. TCP is a means of communication, the word "request" is the concept of a transaction, the HTTP protocol is a transaction protocol, if you send an HTTP request, this statement is no problem. Also often hear the interviewer feedback some of the students interviewed Yun-Wei, the basic TCP three handshake concept is not clear, the interviewer asked TCP is how to establish a link, the interviewer came up to say, if I was a client I sent a request to the server, the server sent a request to me ... This kind of listen to know the basic concept of TCP is not clear. The following is the process of creating a handshake by a TCP that I crawl through Wireshark. (The command line is basically tcpdump, and we'll use this diagram to illustrate the problem later):

Now I see as long as the first 3 lines, this is the TCP three handshake the complete establishment process, the first message SYN issued from the initiator, the second message syn,ack from the connected side, the third message ACK confirmed that the other side of the Syn,ack has been received, the following figure:

But the data is not actually transmitted, the request is data, the fourth report wisdom is the process of data transmission began, careful readers should be able to find Wireshark the fourth message parsing into the HTTP protocol, the HTTP protocol get method and URI also parse out, Therefore, the TCP layer is the concept of no request, HTTP protocol is the concept of transactional protocol, TCP message hosting HTTP protocol requests (request) and response (Response).

Now is the beginning to explain why there is keepalive. Link is established, if the application or upper-level protocol has not sent data, or a very long time to send data, when the link is long no data message transmission how to determine whether the other is still online, in the end is off the line or do not have data transmission, links need not be maintained, This situation needs to be considered in the design of TCP protocols. The TCP protocol solves this problem in a clever way, when more than a period of time, TCP automatically send a data for the empty message to each other, if the other party responded to this message, indicating that the other side is still online, links can continue to maintain, if the other party does not have a message returned, and tried many times after the link is lost, There is no need to keep links.

How to open keepalive

KeepAlive is not enabled by default, and there is no global option on the Linux system to open TCP keepalive. Applications that need to open keepalive must be opened separately in the TCP socket. The Linux kernel has three options that affect the behavior of keepalive:
1.NET.IPV4.TCPKEEPALIVEINTVL = 75
2.net.ipv4.tcpkeepaliveprobes = 9
3.net.ipv4.tcpkeepalivetime = 7200
The unit of the Tcpkeepalivetime is the second, indicating how many seconds after the TCP link does not have the data message to transmit the start detection message; The TCPKEEPALIVEINTVL unit is also seconds, representing the time interval between the previous probe packet and the last probe packet, and the tcpkeepaliveprobes indicates the number of probes.

The TCP socket also has three options and a kernel counterpart, which is set for a separate socket via the setsockopt system call:
TCPKEEPCNT: Covering Tcpkeepaliveprobes
Tcpkeepidle: Covering Tcpkeepalivetime
TCPKEEPINTVL: Covering TCPKEEPALIVE_INTVL

For example, with my system default setting, for example, the kernel default setting of Tcpkeepalivetime is 7200s, if I open keepalive for the socket in the application and then set the Tcp_keepidle to 60, Then the TCP protocol stack sends the first probe message when it finds that the TCP link is idle for 60s without data transfer.

is the TCP keepalive the same as the HTTP keep-alive?

It is estimated that many people at first glance that the problem is often said keepalive is not so, in fact, in the absence of specific TCP or HTTP layer keepalive, can not be confused. The keepalive of TCP and the keep-alive of HTTP are completely different concepts. The keepalive on the TCP layer has been explained. What is the concept of the keep-alive of the HTTP layer? In describing the TCP link establishment, I drew a three-time handshake diagram, TCP after the link is established, the HTTP protocol uses the TCP Transport HTTP protocol requests (request) and response (Response) data, a complete HTTP transaction as follows:

All reader Please note that this diagram simplifies HTTP (Req) and HTTP (RESP), and that the actual request and response require multiple TCP messages. From the diagram can be found a complete HTTP transaction, the establishment of links, the request to send, response received, disconnect the four processes, the early transmission of data through the HTTP protocol to the main text, a request may have all the data to be returned, but, Now to show a complete page requires a lot of requests to complete, such as pictures, JS,CSS, etc., if each HTTP request needs to create a new and disconnect a TCP, this overhead is completely unnecessary, open HTTP keep-alive, can reuse the existing TCP link, Once the current request has been responded to, the server side does not immediately close the TCP link, but instead waits for a time to receive a second request that the browser may send over, usually the browser sends a second request immediately after the first request returns, and if there is only one link at a time, The more requests that are processed by the same TCP link, the more consumption of TCP setup and shutdown that can be saved by opening keepalive. Of course, multiple links are usually enabled to request resources from the server, but after the keep-alive is turned on, it still speeds up the loading of resources. After http/1.1, the default opens Keep-alive, and the connection option is added to the HTTP header field. When set to Connection:keep-alive to open, set to Connection:close to indicate shutdown. In fact, the keepalive of HTTP is keep-alive, which is different from the keepalive of TCP. So TCP keepalive and HTTP keep-alive are not the same thing.

nginx TCP keepalive How to set the

The opening mentions my recent problem, the client sends a request to the Nginx service side, the service end needs after a period of computation will return, the time surpasses the LVS session to maintain 90s, in the service end uses tcpdump grasping the package, The results shown locally through the Wireshark analysis, as shown in the second diagram, are about 90s short of the time stamp between the 5th message and the last message. After determining if the session retention time of LVS is due, I began to look for Nginx TCP keepalive How to set, the first option to find is KeepAliveTimeout, I learned from my colleagues that the use of keepalivetimeout is to turn off keepalive when the value of KeepAliveTimeout is 0, and to indicate how many seconds the link holds when the value of KeepAliveTimeout is a positive integer value. The KeepAliveTimeout is then set to 75s, but the actual test results indicate that it is not effective. Obviously keepalivetimeout can not solve the TCP level keepalive problem, in fact, nginx involves a lot of keepalive options, nginx the usual way to use the following:

From the TCP level Nginx not only to be concerned with the client keepalive, but also with upstream care keepalive, at the same time from the HTTP protocol level, Nginx needs and client care keep-alive, If the HTTP protocol used by upstream, but also care about and upstream of the keep-alive, in general, but also more complex. So after you know the TCP layer's keepalive and HTTP keep-alive, you will not set the wrong keepalive for Nginx. I was dealing with this problem. Nginx has the option to configure TCP keepalive, so I open the Ngnix source code, search in the source code tcp_keepidle, the relevant code is as follows:

From the context of the code I found that the TCP keepalive could be configured, so I went on to find out which option to configure, and finally found that the so_keepalive option for the listen directive could keepalive the TCP socket.

The above three parameters can only be used for one and cannot be used at the same time, such as Sokeepalive=on, Sokeepalive=off or sokeepalive=30s::(to wait for 30s to send a probe message without a datagram. By setting up listen 80,sokeepalive=60s:: After successfully resolving the problem of maintaining long links in LVs, Nginx avoids the use of other high-cost programs. If you encounter a similar problem on a commercial load device, you can also solve it in this way

I've had a problem with my recent job and wanted to write it down, and the scenario is this:

From the above figure can be seen, the user through Client access is LVS VIP, VIP back-end Mount Realserver is Nginx server. Client can be a browser or a client program. In general, this architecture will not cause problems, but if the client side of the request to send the Nginx,nginx back end will take some time to return the results, more than 1 minutes and 30 seconds there is a problem, using LVS as a load balancing device to see the phenomenon is 1 minutes 30 seconds later, The client and Nginx links are disconnected and no data is returned. The reason is LVS default to keep TCP session 90s, more than 90s no TCP message transmission on the link, LVS will send reset message to both ends of the link. The reason for LVs to believe that we all know twos, I know the reasons are mainly two points:

1. Save load Balancing equipment resources, each TCP/UDP link will create a session structure on the load balancing device, if the link is kept open, this session structure information will eventually consume all resources, so it must be released. 2. Another release can protect the back-end resources, if the attacker through an empty link, link to nginx, if Nginx did not do appropriate protection, Nginx will be unable to provide services because of too many links.

This problem is not only on the LVS, before the commercial load balancing equipment F5 encountered the same problem, F5 session disconnected way and LVs a little different, F5 will not actively send reset to the ends of the link, the session disappeared, When a link to send a message again will receive F5 reset, after the phenomenon is sent again the end of the message TCP link state has been disconnected, and the other end is still establish state.

Once you know the reason for the load balancing device, the first reaction is to solve it by opening the keepalive. This should be the end of the problem, but I found that some time there is always someone to mention the problem of keepalive, and even found that due to KeepAlive's understanding is not correct to waste a lot of resources, could have used the application of LVS in the public net subsidence area, or replaced by commercial F5 equipment ( The session disconnect time for the F5 device is a bit longer and should be 5 minutes by default. So I decided to write a blog about the keepalive knowledge I know.

Why should there be keepalive.

Before we talk about KeepAlive, we should first understand the simple TCP knowledge (the knowledge is very simple, the master directly ignores). The first thing to be clear is that there is no "request" on the TCP layer, and it is wrong to hear that a request is sent at the TCP layer often. TCP is a means of communication, the word "request" is the concept of a transaction, the HTTP protocol is a transaction protocol, if you send an HTTP request, this statement is no problem. Also often hear the interviewer feedback some of the students interviewed Yun-Wei, the basic TCP three handshake concept is not clear, the interviewer asked TCP is how to establish a link, the interviewer came up to say, if I was a client I sent a request to the server, the server sent a request to me ... This kind of listen to know the basic concept of TCP is not clear. The following is the process of creating a handshake by a TCP that I crawl through Wireshark. (The command line is basically tcpdump, and we'll use this diagram to illustrate the problem later):

Now I see as long as the first 3 lines, this is the TCP three handshake the complete establishment process, the first message SYN issued from the initiator, the second message syn,ack from the connected side, the third message ACK confirmed that the other side of the Syn,ack has been received, the following figure:

But the data is not actually transmitted, the request is data, the fourth report wisdom is the process of data transmission began, careful readers should be able to find Wireshark the fourth message parsing into the HTTP protocol, the HTTP protocol get method and URI also parse out, Therefore, the TCP layer is the concept of no request, HTTP protocol is the concept of transactional protocol, TCP message hosting HTTP protocol requests (request) and response (Response).

Now is the beginning to explain why there is keepalive. Link is established, if the application or upper-level protocol has not sent data, or a very long time to send data, when the link is long no data message transmission how to determine whether the other is still online, in the end is off the line or do not have data transmission, links need not be maintained, This situation needs to be considered in the design of TCP protocols. The TCP protocol solves this problem in a clever way, when more than a period of time, TCP automatically send a data for the empty message to each other, if the other party responded to this message, indicating that the other side is still online, links can continue to maintain, if the other party does not have a message returned, and tried many times after the link is lost, There is no need to keep links.

How to open keepalive

KeepAlive is not enabled by default, and there is no global option on the Linux system to open TCP keepalive. Applications that need to open keepalive must be opened separately in the TCP socket. The Linux kernel has three options that affect the behavior of keepalive:
1.NET.IPV4.TCPKEEPALIVEINTVL = 75
2.net.ipv4.tcpkeepaliveprobes = 9
3.net.ipv4.tcpkeepalivetime = 7200
The unit of the Tcpkeepalivetime is the second, indicating how many seconds after the TCP link does not have the data message to transmit the start detection message; The TCPKEEPALIVEINTVL unit is also seconds, representing the time interval between the previous probe packet and the last probe packet, and the tcpkeepaliveprobes indicates the number of probes.

The TCP socket also has three options and a kernel counterpart, which is set for a separate socket via the setsockopt system call:
TCPKEEPCNT: Covering Tcpkeepaliveprobes
Tcpkeepidle: Covering Tcpkeepalivetime
TCPKEEPINTVL: Covering TCPKEEPALIVE_INTVL

For example, with my system default setting, for example, the kernel default setting of Tcpkeepalivetime is 7200s, if I open keepalive for the socket in the application and then set the Tcp_keepidle to 60, Then the TCP protocol stack sends the first probe message when it finds that the TCP link is idle for 60s without data transfer.

is the TCP keepalive the same as the HTTP keep-alive?

It is estimated that many people at first glance that the problem is often said keepalive is not so, in fact, in the absence of specific TCP or HTTP layer keepalive, can not be confused. The keepalive of TCP and the keep-alive of HTTP are completely different concepts. The keepalive on the TCP layer has been explained. What is the concept of the keep-alive of the HTTP layer? In describing the TCP link establishment, I drew a three-time handshake diagram, TCP after the link is established, the HTTP protocol uses the TCP Transport HTTP protocol requests (request) and response (Response) data, a complete HTTP transaction as follows:

All reader Please note that this diagram simplifies HTTP (Req) and HTTP (RESP), and that the actual request and response require multiple TCP messages. From the diagram can be found a complete HTTP transaction, the establishment of links, the request to send, response received, disconnect the four processes, the early transmission of data through the HTTP protocol to the main text, a request may have all the data to be returned, but, Now to show a complete page requires a lot of requests to complete, such as pictures, JS,CSS, etc., if each HTTP request is