TCP Series 34-Window Management & Flow Control-8, cache auto-tuning

I. Overview

We have previously described a network with a large bandwidth delay product (Band-delay product, BDP) , which is called a long-fat network (longfatnetwork, or LFN). We imagine a simple scenario, assuming the sending window is 5000bytes, the network RTT is 200ms, then the maximum rate per second is 5000* (1000/200) =25000bytes/s, which is about 24kb/s, you can see that this rate is very low , this is the TCP send window for the transmission rate limit, the actual window size should be at least the bandwidth delay product to efficiently utilize the network transmission capacity. Therefore, for the long-fat network to expand the TCP window size, we have previously described the RFC1323 improved high-performance TCP extension option wsopt, which allows TCP to use a larger window size (approximately 1GB). However, it is difficult for us to know the bandwidth delay product of this network when the connection is established, which requires an auto-tuning (auto-tuning) Algorithm for window size, which can automatically raise TCP reception cache for the long-fat network. The window size is then enlarged. Sometimes this receive cache auto-tuning technique is also known as Dynamic right-sizing (DRS).

Ii. Simple introduction of several configuration parameters related to Linux

Net.ipv4. Tcp_moderate_rcvbuf

When this parameter is set to non-0 and the user does not set the receive cache for the socket via SO_RCVBUF, TCP automatically adjusts the receive cache. However, the maximum number of receive window adjustments will not exceed tcp_rmem[2]. This can greatly increase the transmission rate of the receiving window for networks with long Singo transmission rates.

Net.core. Wmem_max and net.core. Rmem_max

These two parameters determine the maximum send cache values and receive cache values that can be set by the two socket options via SO_SNDBUF and So_rcvbuf, plus two lower limits when set by So_sndbuf and So_rcvbuf. On the ubuntu16.04 are 4608 and 2304, and also in the previous example we introduced the two options to set the receive cache when the actual kernel will set the value of the first 2 bar, if the SO_RCVBUF set the receive cache to 3500, then the kernel will max (min ( 3500*2,rmem_max), 2304) = 7000, and eventually the receive cache is set to 7000.

Net.ipv4. Tcp_adv_win_scale

The 7000bytes cache in the example above is divided into two parts, which is called the kernel cache for the receiving window, which corresponds to the actual received valid TCP data, and the other part is used to store the auxiliary structure (SKB) of these TCP data, which is called the application cache, this is the Linux man The name of the page, the application cache this is not appropriate (Linux man page can also be wrong, for example, the explanation of the Tcp_retries2 parameter is wrong, for this application cache and kernel cache explanation really do not want to vomit slot). The proportions of these two parts can be set by Tcp_adv_win_scale. If tcp_adv_win_scale>0, the application cache takes up Bytes/2^tcp_adv_win_scale, and if this parameter is less than 0, the application cache is bytes- bytes/2^ (-tcp_adv_win_scale). For example , when set to Tcp_adv_win_scale to 2, the cache for window size is 7000* (1-1/2^2) =5250bytes.

Net.core. Wmem_default and Net.core. Rmem_default

These two parameters specify the default send cache and receive cache for the Transport Layer Protocol

Net.ipv4. tcp_wmem and Net.ipv4. Tcp_rmem

Where tcp_wmem This value is composed of three integers, three integers are min, default, Max. Min is the guaranteed value for the minimum receive cache for each socket, even if there is a memory pressure, if the currently allocated receive cache is less than min, the allocated memory can still be allocated. Default overrides the Send cache defaults for the Transport layer protocol specified by the Wmem_default,wmem_default, and default further specifies the defaults for the send cache under the TCP protocol. For example, UDP does not specify its own default value for Send cache, then Wmem_default is used as the default value. MAX Specifies the maximum value for the send cache. Tcp_rmem This parameter is similar to Tcp_wmem.

Iii. Examples of Wireshark

1, Cache auto-tuning this piece is relatively complex in implementation, here is only an example, no longer carefully explain the adjustment process. As shown in the following two images, the client begins writing 40bytes data every 5ms, writes three consecutive times, and then the client writes 5 4096bytes of data consecutively, without pausing between each write. You can see that at the beginning of each write 40bytes data when the server side window size has been 2088, after a large number of continuous write 5 times 4096bytes of data, the server side began to slowly increase the window Size until the end of the 49th Packet window size becomes 43848.

Additional notes:

1, http://kb.pert.geant.net/PERTKB/TCPBufferAutoTuning

2, http://patchwork.ozlabs.org/patch/276802/mbox/

3, the specific receiver cache adjustment algorithm has exceeded the scope of this article, you can refer to Tcp_rcv_space_adjust, tcp_rcv_rtt_update, Tcp_clamp_window, Tcp_grow_window

From for notes (Wiz)

TCP Series 34-Window Management & Flow Control-8, cache auto-tuning