CPU affinity problems encountered by Kthread in Netback

Recently in the upgrade netback, during the testing process, found that the VM full-duplex pressure, rx PPS Volatility is very strong, see the RX Kthread Although the CPU affinity is 0-7 (Dom0 8vcpu), but often run to the physical interrupt of the CPU.

By manually binding the RX Kthread CPU to the other CPUs, Rx PPS is up and stable, and it is obvious that the PPS fluctuation of RX is due to CPU scheduling, which is dispatched to different CPUs.

There is a question, why the CPU scheduling, will be the Rx Kthread Dispatch to si% the highest CPU, load balance mechanism does not play a role?

First look at the code, the RX Kthread implementation is a wait_event, and so on the packet from the network card to call Vif Start_xmit, triggering wake_up. This means that the packet comes up from the NIC and triggers a soft interrupt on that CPU, and then wake_up our Rx Kthread to get up and work.

So does the Rx Kthread have something to do with Wake_up's CPU?

Xiantao Daniel, I looked at a KVM on the VM thread scheduling problem that they discovered very early, https://lkml.org/lkml/2010/4/11/108, is essentially the same problem.

Simple tracking of the code, TRY_TO_WAKE_UP will call Sched_fair.c's Select_task_rq_fair to pick a CPU, as the woken task running CPU. If sched_feature.h inside defines the

Affine_wakeups so want_affine=1, then there is a affine_sd, representing the affinity Scheduling_domain, and then call Wake_affine, which will be on the last run of the CPU and the current Wake_ Up

CPU, perform some load-related comparisons to choose whether to select an idle sibling (select_idle_sibling) based on the PREV_CPU or WAKE_UP CPU

In our scenario, want_affine=1, Wake_affine=1, select_idle_sibling (wake_up CPU), and WAKE_UP CPU is also idle, although the soft interrupt is very high, but no other thread scheduling, Soft interrupts are triggered in the idle context all the time. If the above conditions are met, the new CPU try_to_wake_up get is WAKE_UP CPU. In rare cases, wake_affine=0, the new CPU is still prev_cpu, equals no migration.

Done an experiment, put sched_features.h inside the affine_wakeups = 0, then Want_affine = 0, will not go to the above logic, and finally basically there is no migration, has been running on PREV_CPU.

Without the previous fluctuations, the performance also went up.