Record a multi-core CPU load imbalance (dynamic bind process to specified CPU:TASKSET-PC $CPU $PID)

A question was discussed with a reader friend last night:on a multi-core CPU Web server, there is a load imbalance, in which the load of CPU0 is significantly higher than other cpux, further investigation indicates that the PHP-FPM suspicion is very large. I have previously documented a similar problem with soft interrupts, but in this case we can exclude suspects.
Let's sample the data on a four-core server to see if there is a load imbalance problem:
Shell> mpstat-p all 1 10

CPU%usr%nice%sys%iowait%irq%soft ...%idle
All 17.57 0.03 1.78 0.00 0.35 0.23 ... 80.04
0 43.17 0.00 4.12 0.00 1.41 1.00 ... 50.30
1 9.80 0.00 0.81 0.00 0.00 0.00 ... 89.39
2 9.31 0.00 1.20 0.00 0.00 0.00 ... 89.49
3 7.94 0.10 0.80 0.00 0.00 0.00 ... 91.16

If the meaning of the above command is to run once per second Mpstat, a total of 10 times to take the average, it can be seen that the idle CPU0 is significantly smaller than the other cpux, and most of them are consumed in the user-state usr above.
Let's check Pidstat to see if the PHP-FPM is causing the CPU0 load problem:
Shell> Pidstat | grep php-fpm | awk ' {print $ (NF-1)} ' | Sort | Uniq-c

157 0
34 1
34 2
32 3

It is visible that the PHP-FPM process assigned to CPU0 is more than the sum of the other three cpux. Why is most of the process allocated to CPU0? I have a vague impression that the operating system prefers to use CPU0, but I haven't found any real clues to prove it, if anyone knows, please tell me.
The problem is always solved, since PHP-FPM does not have a CPU affinity (affinity) binding directive like Nginx, then we can use Taskset binding php-fpm process to fixed cpux to solve the problem:
#!/bin/bash

cpus=$ (grep-c processor/proc/cpuinfo)
pids=$ (ps aux | grep "php-fpm[:] Pool" | awk ' {print $} ')

Let I=0
For PID in $PIDs; Do
Cpu=$ (echo "$i% $CPUs" | BC)
Let i++

TASKSET-PC $CPU $PID
Done

Once the script is running, let's take a look at how each CPU load is distributed:
Shell> mpstat-p all 1 10

CPU%usr%nice%sys%iowait%irq%soft ...%idle
All 15.73 0.03 1.61 0.00 0.20 0.23 ... 82.20
0 16.28 0.10 1.62 0.10 0.81 0.91 ... 80.18
1 16.16 0.10 1.51 0.00 0.00 0.10 ... 82.13
2 14.46 0.10 1.71 0.00 0.00 0.00 ... 83.73
3 15.95 0.00 1.71 0.00 0.00 0.00 ... 82.35

Finally average, but it should be reminded that once the number of requests processed by PHP-FPM exceeds the max_requests setting, then the corresponding process will be restarted automatically, and the previous Taskset settings will be invalidated, so we need to add the Taskset script to the CRON in order to remain valid. Configuration, such as auto-set every minute!
In this paper, the PHP-FPM process evenly distributed to the 0,1,2,3 four CPUs, the actual operation can be more flexible, such as we mentioned earlier, the operating system is always preferred to use CPU0, if the CPU0 load is already high, then we might as well the PHP-FPM process allocated to 1, 2, 33 CPUs.

Original link

Record a multi-core CPU load imbalance (dynamic bind process to specified CPU:TASKSET-PC $CPU $PID)