Linux network protocol stack is very independent, up and down through two interfaces, respectively, and the user state and device connected, can also be considered as the North and South interface ... North through the socket interface, south through the Qdisc interface (you can think of the upper Netdev queue, for the receiving interface, NAPI poll queue is another example), whether the socket or QDISC, is based on the queue to manage, that is, Three parts are independent, the socket can only see read and write queue, and not see the protocol stack itself, the socket when reading a data, it takes the data in the queue, as to say this data is who put in, it does not know, is not the protocol stack put in, it does not have to verify.
The socket isolates the user process and protocol stack, and the RX/TX queue isolates the protocol stack and device drivers.
This isolation provides simplicity in programming and design, but is not conducive to performance.
The RPS design of Linux is designed to allow a CPU to handle both the protocol stack receiving process of the packet (soft interrupt kernel thread context, or the soft interrupt processing of any context), and the process by which the user state processes the packet. I say this design is good and bad, if only to improve the cache utilization, then this design is right, but there is no other case, if a CPU at the end of the net RX Soft interrupt processing to push a SKB to a socket queue, and try to wake up the waiting process, So what does it do next? In fact it should go back to the device next, continue to poll the next SKB, but the RPS design is not so, RPS is designed to let the CPU continue to handle the user-state process .... This will inevitably be a process switch and user/kernel switching, although the server CPU cache utilization increased, but the protocol stack processing related CPU cache utilization is reduced. In fact, whether the CPU cache is refreshed after process switching and user/kernel transitions is architecture-related, not that all architectures can produce good results.
Further testing must be done.
I think the best approach is that the user process and the kernel's net Rx Soft interrupts are at different CPU cores, however these two CPU cores share the two level cache or the three cache.
...
The Linux kernel has developed a better solution, which is to break through the above independent three parts, so that the socket directly into the device layer directly poll skb!! Note that this is a poll operation and does not allow the socket to handle the protocol stack process directly. The socket direct poll means that the socket does not read the packet in the queue, not sleep, and then waits for the net RX kernel thread to wake up the packet after it has been queued, instead of directly asking the device: Is there a packet now? If so, I take them directly to the protocol stack without you having to send them. This is a "pull" way, not the kind of "push" way, pull and push the difference is that for the recipient, pull is the same entity, is active, and the push is passive.
This solves the problem that RPS is trying to solve without a perfect solution. This mechanism is called busy poll.
RPS tries to get the soft interrupt processed after the packet is switched to the user process, at which point the soft interrupt will be intermittent, and then the packet is interrupted and then cut back ... busy poll is not so, it directly bypasses the soft interrupt this executor, directly by the socket itself in the execution body to actively pull the packet for processing. Avoid switching problems caused by a large number of task transitions.
I don't know if it is possible to use busy poll to improve performance in the case of forwarding, which requires testing. The above explanation is only ideal, the real situation is that the socket may pull a packet for another socket from the device, even if the packet is only forwarded, not associated with any socket ... Because a packet can be associated with a specific socket only after standard routing and four-tier processing, it is futile and hopeless to expect to find this association at the device driver level! Anyway, control in the user's own hands, by probability, if your device is a large number of packets are forwarded packets, do not open this function, if your process has a small number of sockets processing a large amount of data packets, then open it, anyway, this is only a usage and configuration problems, when to open, And how much the share is set, requires a pre-sampling process.
Early this morning, wrote two essays, so also did not go out, now almost seven o'clock, little and child her mother is still asleep, I am ready to go to work ....
Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.
Linux Kernel 3.11 Socket busy poll mechanism to avoid sleep switching
