Introduction to multiple queue NICs

Article Source: http://blog.csdn.net/turkeyzhou/article/details/7528182

 

Multi-queue Nic is a technology used to solveNetwork I/O QoS (Quality of Service)However, with the increasing bandwidth of network I/O,Single-core CPUUnable to fully meet the requirements of the NIC, supported by multi-queue NIC driver,Bind each queue to different cores through interruptionTo meet the NIC requirements.

Common examples include Intel 82575, 82576, and boardcom 57711. The following uses an Intel 82575 NIC that is used by the company's servers as an example, analyze the hardware implementation of multi-queue NICs and the support of Linux kernel software.

1. Multi-queue Nic hardware implementation

Figure 1.1 is an Intel 82575 hardware logic diagram with four hardware queues.When a packet is received, the same queue is always received for a stream through the SIP, sport, dip, and dport tuples of the hash packet header.. At the same time, the interrupt bound to the queue is triggered.

 

2. Implementation of NIC driver before 2.6.21

The multi-queue feature is not supported by kernel from 2.6.21,One Nic can only apply for one interrupt number,Therefore, at the same time, only one core is used to process the packets received by the network adapter.. 2.1,The protocol stack uses napi to receive packets from various hardware queue.In the net_device data structure shown in Figure 2.2, messages are sent to the NIC through the qdisc queue.

 

 

3. Implementation of the NIC driver after 2.6.21

2.6.21 starts to support the multi-queue feature. When the NIC Driver is loaded, the number of hardware queue of the NIC is obtained based on the NIC model and the number of CPU cores, finally, sum = min (NIC queue, CPU core) is used to obtain the number of NICs queue to be activated (SUM), apply for a sum interrupt number, and assign it to each activated queue.

3.1. When a queue receives a message, it triggers the corresponding interrupt and receives the interrupted core, add the task to the net_rx_softirq queue of the core responsible for collecting packets in the protocol stack (net_rx_softirq has an instance on each core). In net_rx_softirq, call the napi package receiving interface, the packet is received in the net_device data structure of 3.2 of the CPU with multiple netdev_queue.

In this way, each core of the CPU can receive packets concurrently, and it will not be one core that cannot meet the requirements, resulting in reduced network I/O performance.

 

 

4. Bind interrupted

When the CPU can receive packets in parallel, different cores receive messages from the same queue, which leads to disordered packets, the solution is to bind the interruption of a queue to a unique core, so as to avoid disorder. At the same time, if the network traffic is high, the soft interrupt can be evenly distributed to each core to avoid CPU bottlenecks.

 

 

5. Interrupt and joint correction

Some multi-queue Nic drivers are not implemented very well. After initialization, the Tx and Rx interruptions of the same queue in Figure 4.1 will be bound to different cores, in this way, data flows between core0 and core1, resulting in increased data interaction between cores, reduced cache hit rate, and reduced efficiency.

 

 

 

David Miller, head of the Linux network subsystem, provides a script to first retrieve information in the/proc/interrupts file, get the interrupted mask according to VEC in eth0-rx-0 ($ VEC) in Figure 4.1, and

Write the data to the smp_affinity corresponding to the interrupt number 53. Since the eth-rx-0 is the same as the vec of the eth-tx-0, TX implementing the same queue is bound to a single core with the RX interrupt, as shown in 4.3.

 

 

The set_irq_affinity script is located at http://developer.oa.com/tlinux/tools/set_irq_affinity.sh.

 

6. Multi-queue Nic Recognition

# Lspci-vvv

The entry content of the Ethernet controller. If MSI-X & enable + & tabsize> 1 exists, the NIC is a multi-queue Nic, as shown in 4.4.

 

 

Message signaled interrupts (MSI) is an implementation of the PCI specification. It can break through the limit of 256 CPU interrupt, making it possible for each device to have multiple reconnection, the multi-queue NIC driver applies for MSI for each queue. The MSI-X is an MSI array, enable + directive, tabsize is the array size.