New Network Features of Linux kernel 4.4

New Network Features of Linux kernel 4.4

This article has a big question, but I only want to describe some of the features that I have been paying attention to, and will not be too detailed. As usual, I mainly want to help clarify the ideas without analyzing the source code. This is mainly for the day when I suddenly forgot, I could remember my understanding at that time with a glance of ten rows, or I could not understand the details I wrote.

Lockless TCP listener

Starting with syncookie of TCP, if syncookie mechanism can be used well, but not, it will lose a lot of Option Negotiation information, which is crucial to TCP performance. TCP syncookie is mainly used to prevent semi-connected syn flood attacks. A large number of nodes send a large number of syn packets, and no matter what, when the attacked protocol stack receives a syn, a request is created and bound to the request queue of the Listener. This will consume a lot of memory. But think about it, let alone option negotiation. For TCP syn and synack, TCP only needs to find the Listener in three handshakes, you can directly construct a synack package based on the syn package. You don't need to use Listener. There are two ways to remember the information of two handshake packets, the first method is to send the syncookie mechanism to encode and echo back. After the 3rd handshake ack arrives, TCP will decode the ack serial number, construct the sub-socket, and insert the accept queue of Listener, another way is to allocate memory locally, record the information of the connected client, wait for the 3rd handshake packet ack to arrive, find the request, construct the sub-socket, and insert the accept queue of Listener. Before 4.4, a request belongs to a Listener, that is, a Listener has a request queue. Each request constructed must operate on the Listner itself, however, the 4.4 Kernel provides a breakthrough method to construct a new socket based on this request! Insert it into the global socket hash table. This socket only records the lightweight reference of its Listener. When the ack of the 3rd handshake packets arrives, the socket hash table will not be queried, but the new socket constructed when the syn packet arrives, in this way, the traditional logic below can free up Listener: the traditional TCP protocol stack receiving

sk=lookup(skb);lock_sk(sk);if(skisListener);thenprocess_handshake(sk,skb);elseprocess_data(skb);endifunlock_sk(sk);

It can be seen that the sk lock period will be a bottleneck, and all the handshake logic will be processed during the lock period. The 4.4 kernel has changed all of this. The following is the new logic:

sk=lookup_form_global(skb);if(skisListener);thenrv=process_syn(skb);new_sk=build_synack_sk(skb,rv);new_sk.listener=sk;new_sk.state=SYNRECV;insert_sk_into_global(sk);send_synack(skb);gotodone;elseif(sk.state==SYNRECV);thenlistener=sk.lister;child_sk=build_child_sk(skb,sk);remove_sk_from_global(sk);add_sk_into_acceptq(listener,child_sk);filock_sk(sk);process_data(skb);unlock_sk(sk);done:

In this logic, you only need to lock the specific queue in fine granularity and do not need to lock the entire socket. Syncookie logic is simpler, and SYNRECV socket does not need to be constructed at all. You only need to ensure that there is a Listener! This was the 4.4 new feature that I suddenly saw when I went to my restroom on Thursday morning. I was shocked at the time. This was exactly what I accidentally thought of in 2014, but I did not follow up since I had no environment, it is now in mainline, and I have to say this is a good thing. At that time, my idea was to construct a synack based on a syn Packet and ignore the Listner. The information to be negotiated can be saved in other places without binding to the Listner, this frees Listener from its responsibilities. However, I did not expect to construct another socket and insert it into the same socket hash table in parallel with all sockets. I think the logic before 4.4 is simple and clear. Whether it is a handshake packet or a data packet, the processing logic is completely consistent, but 4.4 complicate the code, separated so many if-else... however, this is inevitable. In fact, the request constructed by syn should be bound to the Listener, but the code will become complicated if you think of optimization, but the code will look good if you work hard on the Code itself, but I don't have that capability, and my code is not well written. The concept of Lockless is similar to that of nf_conntrack, But I think conntrack can also be used for the related conn logic.

CPU affinity and REUSEPORT of TCP listener

The accept queue is optimized along with the Lockless TCP Listener! As we all know, a Listener has only one accept queue. In a multi-core environment, this single queue is definitely a bottleneck. How can a high-performance server endure this! In fact, this problem has long been solved by REUSEPORT. The REUSEPORT allows multiple independent sockets to listen to the same IP/Port pair at the same time. This is definitely a good news for today's multi-queue Nic, multi-CPU environment. However, even though the road is wide, there are more lanes. If there are no rules, the performance will decrease, and the congestion level will decrease! 4.4 The kernel introduces a SO_INCOMING_CPU option for the socket. If this option is set to n for a socket, this means that data packets can be inserted into this socket only when the execution stream of the protocol stack logic is processed on cpu n. It is reflected in the Code, that is, the compute_score plus points, that is, in addition to the target IP address, target port, source IP address, and source port, the cpu has become a matching item. As mentioned in patch, this feature, combined with REUSEPORT and multi-queue Nic, must be a good dish!

New stream-based multi-path routing

There is a route cache before it starts. A route cache item is an n-tuples with source information. Each packet will create a cache item after it matches the FIB entry, in subsequent queries, the cache is first searched, so it is based on the stream. However, after the route cache class, multi-path routing becomes packet-based, which will certainly cause disorder in the TCP protocol. For this 4.4 kernel multi-path routing, the source information is introduced in hash computing to avoid this problem. As long as the calculation method remains unchanged, the data of a stream is always hashed to a dst.

Socket route cache with version number

This is not a feature carried by the 4.4 kernel. It is my own idea. Early_demux has been introduced into the kernel to eliminate route searches for incoming traffic from the local machine. After all, the route searches are performed through socket searches. Why not directly search for the socket? Cache route information for the searched result. Enable this option for devices that provide services on the local machine. However, for outbound traffic, there will still be a lot of overhead wasted on route lookup. Although the IP address is connectionless, the TCP socket or a connected UDP socket can clearly indicate a 5-tuples. It is better to store the routing information in the socket. Okay! Many people will ask how to solve the synchronization problem. What should I do if the route table is changed? Should I use notify socket? If you are guided to design an "efficient synchronization protocol", you will lose! The method is simple, that is, two counters-Cache counters and global counters are introduced. The socket routing cache is as follows:

sk_rt_cache{atomic_tversion;dst_entry*dst;};

The global counters are as follows:

atomic_tgversion;

When the socket sets the route cache, it reads the global gversion value and sets it to the cached version. When the route changes, the global gversion counter increments. If the cache counter value is the same as the Global Counter value, it is available. Otherwise, it is unavailable. Of course, dst itself must be protected by the reference counter.