OpenVPN Optimization-giant frame and openvpn Optimization
Over the past few days, I have been busy with the performance problem of OpenVPN. This is really an old problem. I have been repairing and completing it for several years until the implementation of multi-thread multi-processing, it solves the server throughput problem in server mode, so that multiple CPU cores can be fully utilized. But there is no good solution for client optimization.
Perhaps, the rough style is very suitable for server-side optimization, but the client-side optimization needs to focus on the detailed details. However, I was a person who forgot his mind, but I was not very suitable for doing fine work. However, I used to sew a fashion handbag with old jeans, which was not exactly the same in the table. For OpenVPN client optimization, I encountered the term "giant frame.
The results are good. The test results are still satisfactory. Record some ideas but do not record technical details. This is to allow yourself or others to see this article later, what are the advantages of knowing that such a thing cannot be used directly? This will allow you to repeat your ideas, instead of copying commands or code. The code or command you wrote may be irrelevant to yourself a week later. After half a year, you will not understand it... but the idea is permanent. I still remember a short article I wrote about the early maturity of the baseline when I was in the fourth grade...
I don't want to drink recently, because I don't have time to wait. After drinking, I will go to bed early, and I can't do anything. In the evening, it's better to read history books and write blogs than to drink.
Ethernet is a bit of a historical burden. Since its birth, Ethernet has maintained a downward compatibility. compatibility is the most important term in the computer era, it can be comparable to IA32 and Win32 APIs, satisfying investors' psychology and facilitating consumers. However, it is almost possible to maintain compatibility with the technology itself.
In the 10 M Ethernet era, there are rules for the frame length. The maximum frame length and the minimum frame length can be calculated based on the physical characteristics of the line, the maximum frame length is 1500 bytes. Therefore, this 1500 is the default setting for an Ethernet MTU. In the 10 m age, the length is definitely not short. In fact, it is the maximum length that can be sent, the upper-layer data that exceeds this length must be sliced at the IP layer and then reorganized at the receiving end. This process requires more processing time and consumes resources. To avoid IP fragmentation, the MTU of the transmission media must be perceived by the protocol at the upper layer of the IP address. For TCP, This is the MSS negotiation. I will not discuss the details. After MSS negotiation, you can ensure that the length of data sent by TCP is smaller than the MTU of the media. For UDP, you need to manage the length of the datagram by the application layer or dynamically adjust it through the MTU discovery mechanism. In any case, IP address sharding can be avoided.
In the next section, I will describe the details of MTU. Now we know that it is 1500 compatible. In fact, in the 10M age, this 1500 was determined by electrical characteristics, however, in the 10G age, Ethernet had undergone tremendous changes, and 1500 was no longer a hurdle for electrical properties to be crossed. Therefore, the rest of 1500 was only a software definition, you can set it to 15000, but even so, as long as the MTU of a route through the data path is 1500, IP fragmentation will occur. This is a historical burden. It can send longer data, but cannot, because IP fragments may be generated when the data is sent, however, the processing and restructuring of IP fragment may offset the space and time savings caused by large data frames.
To avoid IP fragmentation, the network adapter is always expected to send the smallest data. However, to maximize the data packet processing efficiency, the network adapter is always expected to send the largest data. This is a conflict, compensation calculation trade-offs are required. We know that each group of the group Exchange Network carries metadata. Because the protocol stack is layered, multiple protocol headers at different layers must be encapsulated for each group.
Group exchange network, which is another form of transmission except circuit exchange, but you can reconcile the two in another way. If you encapsulate an entire data stream into a group, that is circuit switching. The data stream has only one group, and one group can have only one path. This path exists objectively before grouping transmission; if you split an entire data stream into multiple groups, it is group exchange. For each group, the routes through the network may be different, and within a period of time, A passage may be grouped by different data streams, which is embodied in the reuse of statistics.
If the group is too small, IP sharding can be avoided, but the same data stream requires too many data groups until the processing is complete, so more data space will be consumed on metadata, at the same time, more processor time is consumed in routing and Switching Based on protocol metadata. If the group is too large, although there will be less space and time consumption on protocol metadata, however, there will be overhead for IP sharding. Note that IP sharding is required after the Data grouping exceeds a certain length on the group switching network, which is determined by the physical electrical characteristics of the line, if you want to transmit ultra-long data groups, electrical faults, such as pulse malformed. Therefore, the maximum data length that can be sent to avoid the whole IP segment is the minimum value of MTU for each segment of the whole process.
For compatibility purposes, 1500 is the default MTU setting for many Ethernet cards, however, manufacturers cannot turn a blind eye to the features of high-end NICs such as 1G/10G, cat5e/cat6 twisted pair wires, and optical fiber cables. Therefore, the configuration interface for MTU is retained, the MTU value of the network card is determined by the user, which is over bytes of the Ethernet frame. to distinguish it from the original standard Ethernet bytes, it is called a giant frame and the name is quite scary, in fact, there is no such thing as a giant. You can set the MTU of your Nic to exceed 1500, but how much can it reach?
Standardization is another tricky issue, because in the switched Ethernet environment, the characteristics of the line can no longer only be passed through the NIC and a single standard cable (such as the early coarse and fine cables) it is closely related to switch configuration, twisted pair Cable category, quality, Nic self-negotiation configuration, and duplex mode. Therefore, although a giant frame occurs, it cannot be controlled, if a giant frame is sent, the narrow router with a small MTU will not talk about the IP segment, but may also affect the receiving of data frames at the peer end, because the cables connecting the two devices are probably not a single cable, even a single cable may have different Nic features at both ends, this will lead to different electrical characteristics of the receiving and receiving devices, resulting in data frame reception failure.
In any case, giant frames are consistent. Without 1500 as the bottom line, the standardization process of giant frames will be much faster.
Using OpenVPN as a medium, we have mentioned so many things that seem unrelated to OpenVPN optimization. If you still do not understand the process of OpenVPN, please refer to the structure diagram of OpenVPN in my previous article. In the figure, I regard OpenVPN as a medium. The data is sent from the tap Nic to the character device, which enters this special media. After the data is encrypted, it is sent to the peer end and decrypted and written to the character device, then it is received by the peer tap Nic, which goes out of this special media.
Since it is a medium, it must have its own "insurmountable electrical characteristics. This electrical feature tells people how it transmits data frames. For OpenVPN, its electrical feature is to encrypt and decrypt data. In specific implementation, OpenVPN encapsulates a so-called "leading pulse" before the data frame sent from the tap Nic, just like the leading pulse when the data frame is transmitted on the copper line ", this is the protocol header of OpenVPN. Let's take a look at the transmission overhead of this special OpenVPN media. The overhead is divided into three parts: space overhead, time overhead, and system overhead.
In terms of space, the OpenVPN protocol header undoubtedly occupies a certain amount of data space. If the MTU of the tap Nic is 1500, add the length of the OpenVPN protocol header L, the maximum Data Length of the OpenVPN data channel will be 1500 + L. We know that this data is transmitted to the physical Nic as a buffer through a UDP (not to mention TCP) socket, if the MTU of the physical network card is also 1500, the maximum length of the total length of 1500 + L + UDP/IP header must exceed 1500, which will lead to IP fragmentation on the physical network card, after the data arrives at the peer OpenVPN, IP segments must be reorganized first, and then the data enters OpenVPN to unencapsulate and decrypt the data... if the MTU of the tap Nic is too small and far smaller than the MTU of the physical Nic, although the OpenVPN header is added, the UDP/IP protocol header may not exceed the MTU of the physical Nic, however, the load rate will be greatly reduced, because for data streams with the same length, there will be a large amount of data space wasted on the OpenVPN protocol header, UDP/IP protocol header, otherwise, if If the MTU is too large, the load rate will be greatly increased, but IP fragmentation will be triggered. After analyzing the time overhead and system overhead, I will draw a conclusion that the overhead of IP sharding and restructuring of modern high-end NICs can be ignored.
In terms of time, the encryption/decryption, HMAC, compression/decompression of OpenVPN consume a lot of CPU time. After measurement, the efficiency of frequently encrypted packets will be less than the efficiency of one-time encrypted packets, the latency efficiency will not change much, but for throughput, the efficiency of big package encryption and decryption will be significantly improved. Although this detail is not affected by the cache, it is related to the CPU pipeline processing, and the details are not described much, especially on the RISC processor, which is more efficient.
In terms of system overhead, because OpenVPN needs to interact with the tap virtual Nic through the tap character device and through socket and peer interaction, the system calls read/write, sending/recv will lead to switching between user and kernel states, and the number of such switching will decrease as the number of data groups increases.
Now it is clear that the MTU of the tap virtual network card is set to super large!
Are we not afraid of IP sharding caused by too much data sent by the socket of OpenVPN and peer communication? Yes, we would rather let it slice the IP on the local machine, nor send it out through the giant frame of the physical network card. First, I don't know if it can be correctly received by the device at the link peer end, second, I do not know the processing efficiency of the device that splits the Data Group carried by the giant frames in the whole path. However, I can still use this device. It has an Intel825XX Gigabit card with the TSO function, that is, TCP Segment Offload (similar to UFO, LRO, MS chimney offloading, etc ), using hardware for IP sharding, we can imagine that the processing performance of such cards is super. What is different from the intermediate routing device is that I know about my devices, the peer OpenVPN also has a hardware configuration similar to that of the NIC (What about sending a giant frame? We recommend that you use the hardware Offload function for local sharding. I think you can weigh it out.) Therefore, the overhead of IP sharding can be ignored.
In the optimization implementation, we recommend that you set the MTU of the tap Nic to a super large value. The premise is that you have to send such large data packets from the tap Nic, if the two OpenVPN endpoints are only the edge nodes of the two networks, you cannot expect that the data must be sent out of a giant frame. Maybe the MTU of the Data endpoint is only 1500, therefore, no matter how the data reaches the tap Nic, there are only 1500 bytes at most (if the data is forwarded by an MTU with only 500 of the routers in the middle, it will be split, and the length of each segment is shorter ), therefore, it is recommended that the endpoints of the data are sent with giant frames.
What if the data endpoint sends out a giant frame but is split before reaching the OpenVPN node's tap NIC with a super large MTU? Well, in software, you can use Linux to load an nf_conntrack_ipv4 module. because it depends on the defrag module, any part of the datagram file will be reorganized before the forward to the tap Nic. Or, mine the automatic reorganization mechanism of the NIC hardware (as far as I know, it is easier to restructure the data sent to the local machine in the hardware chip ).
Never expect to combine multiple short IP packets into a long IP packet, especially for UDP data packets, because this will confuse the boundaries of UDP data packets, in this case, semantic errors occur at the application layer. The IP itself does not handle sharding and reorganization. It can identify which parts belong to the same group, however, once reorganized, the parts cannot be sharded again as they are before the reorganization. The basis for the sharding is only MTU! Just as you can mix N heaps of beans, but you can no longer separate them. For TCP, although this is not a problem, you must perform complex processing, such as maintaining a hash table, and then add all IP packets (non-partition packets) belonging to the same quintuple) form a new IP packet of the appropriate length, and track the total length information at the same time, cannot exceed the MTU of the tap network card. This algorithm must be efficient; otherwise, it will offset the benefits of combined processing of large groups. Facts have proved that it is difficult to implement this algorithm either in the kernel or in OpenVPN itself.
For OpenVPN in P2P mode, it is a good idea to prepare a box with the same size as the MTU of the tap Nic. In fact, it is a new layer of encapsulation, because I know that no matter what data packets are sent to the unique peer as long as they are transmitted through the tap Nic, it is no longer necessary to distinguish the streams identified by the quintuple. When a small IP packet arrives, it is inserted into the box. If the box is full, it is covered and thrown into the tap Nic. This is a good idea, but there is nothing to do with the server mode. In fact, for the server mode, you need to prepare N such boxes. N is equal to the number of OpenVPN clients. Before each IP packet enters the server's tap Nic, the first step is to find the box you want to enter, the multi-queue tun Nic I implemented has already achieved this search.
To achieve higher space efficiency and time efficiency, we recommend that you use the tun mode instead of the tap mode, although complex iroute commands need to be configured in the network-to-network deployment mode (after all, IPv4 does not have to be forced to use automatic configuration, in fact it does not! DHCP is an extremely poor protocol !).
Finally, let's talk about the MTU modification of TAP-Win32 Nic on Windows platform. If you right-click the properties of the TAP Nic and configure any value with the MTU value greater than 1500, an error message is displayed, indicating that the MTU value cannot exceed 1500! But in fact, you can write commands such as tun-mtu 9014 in the OpenVPN configuration file, and no error will be reported, but in actual operation, it does not use the MTU 9014, resolutely (this is not a word of "still", but an absolute one !) It's 1500! So how can I configure the MTU of the TAP Nic as the so-called "giant frame" size? The answer is to use netsh.
Run
Netsh interface ipv4 set subinterfaces "TAP..." mtu = 9014 sore = persistentThis command changes the MTU of your TAP Nic to 9014.
Netsh interface ipv4 show subintView. But now you can see that the properties of the tap nic are still 1500! In any case, it can indeed send a giant frame with a length of 9014Byte. I don't want to talk anymore! In short, Microsoft's interfaces are not easy to use, and they prefer to complicate simple problems. If you want to change the MTU of the TAP, you must not only modify the configuration file, but also use netsh to "actually modify it "!
Compensation for IP sharding overhead and protocol header overhead is a conflict between the two. The other is to construct a giant frame of the tap network card. It is first reorganized in the tap, and then encrypted and encapsulated, and then resliced in the physical network card. The weight is light, and the amount of traffic is self-generated.
The spotlight lights light the stage, but it does not drive away from loneliness. Maybe it is dark to close your eyes if you are a young man with a strong new words.
You had to walk for an hour before March 13, 100. Now you still need an hour to go to work, but in the car, the development of motivation always keeps you farther and farther away from your goal.
Which is a compensation. Come out and try again later...
