Non-balanced handling of NIC interruptions and nic interruptions

Non-balanced handling of NIC interruptions and nic interruptions

When the data volume is large, hard interruptions and soft interruptions may cause bottlenecks.

The network adapter receives data packets and generates an interrupt signal from the network adapter. The CPU copies the network data packets to the kernel, processes the protocol stack, and finally transmits the data to the user space, however, the hardware interrupt processing only involves copying data from the NIC, and the protocol stack processing is handed over to the Soft Interrupt Processing. Therefore, when hard interrupt and Soft Interrupt are concentrated on cpu0, it will bring a burden to scheduling (concentrated on other CPUs will also cause performance bottlenecks. Here is an example of cpu0, this is because the CPU 0 has the largest impact ).

We can see that eth0 eth1 eth2 eth3 has four NICs

Before stress testing

After stress testing

Compare the traffic before and after, it is known that eth2 is used

The interrupt number is 72-80.

2. Bind

Echo 9>/proc/irq/72/smp_affinity_list

Echo 10>/proc/irq/73/smp_affinity_list

Echo 11>/proc/irq/74/smp_affinity_list

Echo 12>/proc/irq/75/smp_affinity_list

Echo 13>/proc/irq/76/smp_affinity_list

Echo 14>/proc/irq/77/smp_affinity_list

Echo 15>/proc/irq/78/smp_affinity_list

Echo 16>/proc/irq/79/smp_affinity_list

Echo 17>/proc/irq/80/smp_affinity_list

 

Nic interrupt number binding can distribute hard interruptions, but this is not thorough enough, because hard interruptions are concentrated on the cores of the above configuration, and soft interruptions work after the hard interruption ends, in the same core, one core also runs both hard interrupt and Soft Interrupt, which will greatly reduce the processing speed of hard interrupt and Soft Interrupt, and cause great performance bottlenecks, if the interruption is not completed, requests cannot be distributed to haproxy processes.

The Soft Interrupt can be balanced to each core through rps and rfs.

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus 24 core, so set fffff, scattered to each core

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

Echo ffffff>/sys/class/net/eth2/queues/rx-0/rps_cpus

 

Echo 4096>/sys/class/net/eth2/queues/rx-0/rps_flow_cnt 32768/N, N is the number of queues for Nic multi-queue

Echo 4096>/sys/class/net/eth2/queues/rx-1/rps_flow_cnt

Echo 4096>/sys/class/net/eth2/queues/rx-2/rps_flow_cnt

Echo 4096>/sys/class/net/eth2/queues/rx-3/rps_flow_cnt

Echo 4096>/sys/class/net/eth2/queues/rx-4/rps_flow_cnt

Echo 4096>/sys/class/net/eth2/queues/rx-5/rps_flow_cnt

Echo 4096>/sys/class/net/eth2/queues/rx-6/rps_flow_cnt

Echo 4096>/sys/class/net/eth2/queues/rx-7/rps_flow_cnt

Echo 32768>/proc/sys/net/core/rps_sock_flow_entries