Let's talk about the TCP Nagle algorithm and the TCP_CORK option.

Event playback uses OpenVPN to transmit virtual desktop traffic. The terminal screen is obviously refreshed frame by frame, and the network environment is a 10-Gigabit LAN.
Analysis 1. First, change OpenVPN to the TCP mode, because the difference between TCP and UDP in the LAN environment is not big, it will not cause the re-transmission overlay problem. The advantage of TCP is that it can be used for arbitrary analysis, because its algorithm is extremely complicated. If it is used for UDP, it is too simple and there is no better adjustment;
2. The analysis process is not the purpose of this Article. Just give the results. Reduce the sending/receiving buffer to 2 times the MTU, reduce the MTU to less than 1000, and add the tcp-nodelay option to OpenVPN. The result is better, but the screen will be refreshed and the solution will continue, final confirmation of this issue is not fixed, because the virtual machine where the virtual desktop is located does not have a graphics card driver installed... (TMD, my most annoying God in the sales eyes, told me that I can solve some problems that do not belong to my category. For example, if I don't even have a signal, I can ensure that my mobile phone is smooth)
TCP. I understand TCP because I am tired of it! If you do not understand it at all, you are not qualified. TCP has a lot of parameters to adjust, and it's just a waste. I always think that the packet capture tool is used to analyze data, not TCP behavior, can you capture a packet and tell me if the Nagle algorithm is enabled? Can you analyze the TCP timestamps caused by packet capture on the client and the connection is rejected? To learn about a protocol, you still need to understand the design principle and the design and modification history of the Protocol so that you can know why it is like this and why it was caused by it. To be honest, sometimes it is good to look at the code, but it is easy to look at the code into a dead end, such as some non-standard things, once entered into the code, people will mistakenly think that it is a standard thing, then, when looking at other different implementations, the first impression will be your perception...
In short, TCP is very difficult and annoying. I have always thought that TCP is well designed, but its development is messy.
Network Utilization and a large number of packets many people regard the purpose of the Nagle algorithm as"Improve Network Utilization", In fact, the so-called"Improve Network Utilization"It's just a Side effect (Side effect...). The main idea of the Nagle algorithm is"Avoid sending large packets". The Nagle algorithm does not stop sending packets. It only blocks sending packets.LargePacket! It is true that sending a large number of packets reduces the network utilization. However, sending a small number of packets must also reduce the network utilization, why don't we stop sending packets directly? Whether it is a large number of packets or a small number of packets, or even a small packet will not be sent, this is the correct solution to improve network utilization! Yes, TCP_CORK does this. Some people say that the CORK options are Nagle enhancements. In fact, they are completely different, and their original intentions are different.
The original intention of the Nagle algorithm: avoid sending a large number of packets and prevent packet flooding on the network,Ideally, only one packet can exist on the network at a time for a TCP connection.. It is more end-to-end optimization.
The original intention of the CORK algorithm: to improve network utilization,Ideally, you do not need to send packets. Only packets that are full or not sent are allowed..
There has always been a misunderstanding about interactive applications, that is, the Nagle algorithm will introduce latency for interactive applications. We recommend that you disable the Nagle Algorithm for interactive applications. In fact, it is the interactive application that introduces a large number of packets. The Nagle algorithm is used for interactive applications! It is inevitable to introduce some latencies. After all, there is a price for everything, but it solves the problem that interactive applications generate a large number of packets,The real problem itself cannot be ignored because of the price paid to solve the problem as a new problem!
For the operation process of the TCP Nagle algorithm and the delayed ACKNagle algorithm, see Wiki. It reduces the sending of a large number of packets, and is actually a packet-based stop-and other protocols. Before the sent package is confirmed, the sender can accumulate the application data during this period, so that the size tends to increase. This is an effective way to avoid confusion window syndrome. Please note that,The confused window refers to the confusion of the receiving end, rather than the confusion of the sending end.The receiving end has to notify itself of the size of the Receiving Window, regardless of whether it will generate a large number of packets on the sending end. However, the sender can be confused. If you notify you, I will not send it. If you are confused, I will not be confused. If you keep releasing small numbers, I will ignore it. I have my own method, it is not sent until the ACK that has sent the package is received.Obfuscation and resistance of Nagle Algorithms. There are two ways to treat confused window syndrome:You are confused."Method, that is, the Nagle algorithm method mentioned above, and the other method is"Obfuscation of the treatment ReceiverOne of the mechanisms is latency ACK (there are other mechanisms, such as not sending small window notifications ). It can be seen that both methods are trying to reduce the number of packets sent. The two methods solve the same problem in the same way. For the sender, ignoring the small window announcement at the receiver is equivalent to saying that the packet is not sent immediately, the packet can accumulate time into a large package. For the receiver, the delay ACK can delay the ACK sending time, and then the delay window announcement. During this period, the receiving window has the opportunity to zoom in further (due to application processing. The two methods are simple, but once they are used together, the situation will be very unfortunate! Because...
The Nagle algorithm and latency ACK act on packets in the opposite direction and the validation packet for this packet. Therefore, their forces are different and the result is that no one can send packets. Just like a rope tied to two frogs, no one else can run it! The key point is that packet sending relies on ACK. However, delayed ACK blocks ACK's instant sending and forms a deadlock. OriginallyTo reduce the number of packets on the network(Emphasize the purpose of Nagle algorithm and delay ACK again. Note that confused window syndrome is only one of the reasons for packet flooding on the network!), But a large amount of latency is introduced manually!
Here is a general explanation. The packet sending of the Nagle algorithm depends on the fast confirmation of the packet sent by the receiver. Therefore, the receiver should work hard in the opposite direction to delay ACK, that is, fast ACK; in contrast, if latency ACK is enabled on the receiving end, the sending end should continuously send data packets, whether it is large packets or small packets (without considering the effect of a slight ACK ), because the sender cannot expect normal ACK packets from the receiver, that is, the sender should disable the Nagle algorithm. The idea behind the above explanation is that data packets and ACK packets are related,The force should be pushed in one direction, and the other side will be pushed. If both sides are pulled, the force will be offset and the force will be deadlocked.. Unfortunately or sadly,The Nagle algorithm and the latency ACK mechanism are both "pull schemes"! Rowing people know that the paddlers have two seating la S, either in the same direction or in different directions. The latter and TCP packets are similar to ACK, you must draw in one direction, though they are in the opposite direction.
The impact of programming models is on the Nagle algorithm Wiki homepage. There is a saying:
In general, since Nagle's algorithm is only a defense against careless applications, it will not benefit a carefully written application that takes proper care of buffering; the algorithm has either no effect, or negative effect on the application.
Visible Programming Model for"Reduce the number of packets on the network"Influence, meaning,The Nagle algorithm is a targeted Optimization-for interactive applications, it is not a universal standardIf you want to have a better solution, don't count on it, or the application can solve it by itself! If you want the Nagle algorithm to reduce the number of packets on the network without introducing significant latency, there are requirements on the way TCP data is generated. Interactive applications are their initial targets. The Nagle algorithm requiresData must be "Table Tennis"That is to say, the data stream has a clear boundary and one time, similar to human-computer interaction, such as telnet, a remote terminal login program. The data is typed by a person from the keyboard, the boundary is basically a press key. One time is input echo and processing echo. In the preceding scenario, the Nagle algorithm ensures that all sent packets have been confirmed before the next packet is sent. Again, we can see that the Nagle algorithm does not prevent packet sending, it only prevents sending a large number of packets.
In other words, the so-called "ping-pong" mode is the "write-read-write-read" Mode-man-machine interaction mode, however, for the "write-read" (many request/response mode Russian C/S services are like this, such as HTTP)-program interaction mode stated in the Wiki, the consequences of Nagle algorithm and delayed ACK tug-of-war will be magnified. As I have already said in the "TCP Nagle algorithm and delayed ACK" section, the two cannot be mixed, we already know this, but it is just explained in terms of the principle of TCP. This section describes the same problem, but focuses on the programming layer, however, it is not as simple as setting TCP_NODELAY with setsocket or turning off the latency ACK.
There is a good article "TCP_CORK: More than you ever wanted to know". The article says that the Nagle algorithm is effective for applications where data comes from user input, however, for the data generated by applications using stream oriented protocols, the Nagle algorithm purely introduces latency, which I highly agree with, because for people, TCP login to a remote computer in Russia is a processor, users want their operations to immediately display the results. The mode is write-read-write-read. However, the data generation logic of a program is not as fixed as that of human-computer interaction, therefore, you cannot assume that the program performs network IO according to any sequence, and the Nagle algorithm is related to the sequence of Data IO. In fact, even if the acceptor does not enable the latency ACK, The Nagle algorithm is applied to the write-read sequence, which means that additional latency is introduced for no reason.
Is it so complicated? The author did not propose how to solve the problem by programming, but the Wiki page of The Nagle algorithm mentioned"Do not rely on the so-called algorithms built in TCP."To optimize TCP behavior. REFERENCE The Nagle algorithm Wiki statement again:
Another thing In general, since Nagle's algorithm is only a defense against careless applications, it will not benefit a carefully written application that takes proper care buffering; the algorithm has either no effect, or negative effect on the application.
This sentence caused me to think about another thing. It happened last week. As the company migrated everything in disorder and was unwilling to work, a network processing accelerator card vendor visited, the other party claimed that their protocol stack was implemented in the user State to improve performance. At first, I sneered at this point. However, combined with my analysis and subsequent thoughts, I found that this was a breakthrough:
When everything depends on the kernel, we may be wrong! The kernel is not controllable. Why not do it yourself. The kernel provides basic services on the infrastructure. providing high-end private customized services will drag the entire kernel down! If the protocol stack is implemented in the kernel (the macro kernel style), you can manage the TCP buffer in the user State to remove the dependency on the complex TCP algorithm. If the protocol stack is implemented in the user State, so we can implement a better Nagle-like algorithm!
I am going to write another article about this. Now I will go back to TCP_CORK: More than you ever wanted to know.
Back to "TCP_CORK: More than you ever wanted to know", the author mentioned the meaning of write, which is"Everything is a file". Since everything is a fileDefine a set of file IO operations for all IO operations.We know that,UNIX initially divides file IO into two types: block device IO and character device IO,This viewpoint or philosophical concept is epoch-making. It brings the simplicity of programming and a refreshing implication for programmers, at that time, neither the TCP/IP standard nor the BSD socket were proposed. When the network protocol stack became a standard, as an IO type, does network I/O belong to the block device I/O or the character device I/O? At that time, there were two ideas: one of them was to separate the network IO from the file IO into a level I/O type, but eventually the BSD socket won, network I/O is also a device I/O. However, the problem is: Does network I/O belong to block device I/O, or should I belong to a character device I/O?
Before answering this questionDefine a device model for network I/O,No device model. What about Device Classification? Finally, the device model of network I/O is defined as packet. As the author said, the device corresponding to network I/O is the network packet (packet ), your application writes network data to a device through the standard file I/O interface write, which is packet! NowThe remaining question is not how packet is flushed to wire, but how to generate this packet.If the data can be accumulated in a buffer and a packet is generated, it is a block device. If the data is one, a packet is generated, which is a character device.Unfortunately, neither the BSD socket nor the protocol stack stipulates whether this buffer exists.Therefore, network I/O, as an alternative type, is independent of block device I/O and character device I/O. Therefore, we can say that network IO or INET socket IO can define a buffer as the buffer from the socket file IO Interface to packet (as a device) or not. What does the definition depend on? It depends on how you explain the network! John Nagle invented the Nagle algorithm from the perspective of "packet flooding will cause network paralysis", while others resisted the Nagle algorithm from the perspective of "buffer packet will introduce latency, this battle is actually meaningless. The key fuse is that the idea of "Everything is a file" has encountered network I/O embarrassment.
Can we answer this question? Should I have a buffer between data and packet? It depends on whether packet is a character device or a block device. The answer to this question depends on the network and the data receiver. For TCP, the packet is a character device in the ideal situation of the long fat pipe, for programs that use telnet to log on to a slow network, packet is a block device. The existence of a buffer depends on many factors rather than one. It is reasonable to exist. The Nagle algorithm exists, but it is not abusive. The end of this section, for example, is controversial when the East and the east of the Yangtze River are rolling over a feasible ship along the river to cut off water storage and generate electricity ....
As the establishment of a dam is controversial, there is also a debate on whether to buffer network data. Therefore, the implementation of the Nagle algorithm does not have a standard, even a de facto standard, not to mention any industrial standards. Therefore, each operating system has its own implementation, and its own style exists in implementation. Linux is no exception. When sending data, there will be a tcp_nagle_check function to determine whether the data buffer should be used. Let's take a look at the annotations to understand its implementation purpose:
/* Return 0, if packet can be sent now without violation Nagle's rules:
* 1. It is full sized
* 2. Or it contains FIN. (already checked by caller)
* 3. Or TCP_NODELAY was set.
* 4. Or TCP_CORK is not set, and all sent packets are ACKed.
* With Minshall's modification: all sent small packets are ACKed.
*/
It is worth mentioning that the TCP_CORK option and the last Minshall's modification. Let's take a look at TCP_CORK first. Many people will associate this option with the Nagle algorithm, in fact, their relationship is not very clear. Strictly speaking, TCP_CORK is introduced to improve the packet load rate. We know that the packet load rate is very small, so it seems like TCP_CORK is related to the Nagle algorithm. Actually,The CORK option improves the network utilization, because it directly disables packet sending (again, the Nagle algorithm does not prohibit packet sending, but only disables sending of a large number of packets).
The implementation of the CORK option is still different from what is mentioned in the document. For example, when the TCP sending window detection package times out, the buffer package is sent, let's take a look at its annotations:
/* A window probe timeout has occurred. If window is not closed send
* A partial packet else a zero probe.
*/
The idea behind it is very simple. Anyway, it also needs to send a probe package, and it cannot be run. The flat white only sends a probe package, and the load rate is too low. Can I take something with me? If allowed, include the buffer packet, although they do not have the buffer to a full size packet. If we say that some students only have full size, they will be very entangled. In fact, the true intention of CORK is to increase the load rate and improve the network utilization. CORK is just an option for Linux implementation.
Finally, let's take a look at the so-called Minshall's modification. What correction has this guy made to the Nagle algorithm? Although the Nagle algorithm requires that the sent packet be confirmed before sending packets, according to the original intention of the Nagle algorithm, it only avoids sending a large number of packets, so why not ignore the confirmation of the big package directly? That is to say, you only need to confirm the sent packet to continue sending the packet. This is indeed an optimization, but it will also bring some additional effects, the Minshall algorithm ensures that only one packet is on the network, but the Minshall algorithm may also send packets as suddenly as a machine gun (as long as the ACK continues to grow ), according to the original intention of reducing the number of packets, it does send only one packet at a time, but the sending interval is short...
Note that TCP is too complex, and anyone's analysis is one-sided. You can identify the vulnerability, because for TCP, parameter adjustment of any of its features will be applied to other features, the long fat pipeline is ideal, but in some cases it is not the best. The Nagle algorithm sometimes needs to be disabled and sometimes needs to be enabled... alas, TCP, TCP