Linux tips: bind Hardware/processes to different CPUs under multiple cores (1)

Hardware interruptions frequently consume CPU resources. If there is a way to allocate a large number of hardware interruptions to different CPUs (cores) under multi-core CPU conditions) processing clearly balances performance well. Currently, multiple CPUs, multiple NICs, and multiple hard disks are not supported on the server. If the NIC can be interrupted, one CPU (core) is exclusive) if the disk IO interrupt exclusive one CPU, it will greatly reduce the burden on a single CPU and improve the overall processing efficiency. I received an email from a netizen the day before yesterday about SMP IRQ Affinity. The following operations are performed on the SUN FIre X2100 M2 SERVER + 64-bit CentOS 5.5 + Linux 2.6.18-194.3.1.el5.

What is interruption?

The definition of "interruption" in Chinese textbooks is too rigid. Simply put, each hardware device, such as a hard disk or network card, is used) both need to communicate with the CPU in some form so that the CPU can know what happened in time, so that the CPU may put down the tasks in hand to handle emergency incidents, when a hardware device actively interrupts the CPU, it can be called a hardware interruption. Just like QQ interference when you are working, a QQ Shake can be called an interruption.

Interruption is a good way to communicate between CPU and hardware, and another method is called polling. This means that the CPU can periodically query the hardware status and then perform corresponding processing, it's like checking QQ every five minutes to see if anyone is looking for you. Is this a waste of CPU time? Therefore, interruption is an active hardware method, which is more effective than CPU polling.

Well, there is another problem here. Every hardware device is interrupted. How can we differentiate different hardware? How do I know which one is from the hard disk and which one is from the network card when the device is interrupted at the same time? It's easy. Isn't every QQ number different? Similarly, each hardware device in the system is assigned an IRQ Number. This unique IRQ Number can be used to distinguish between James and James.

In a computer, interruption is an electrical signal generated by hardware and directly sent to the interrupt controller, such as 8259A). Then, the interrupt controller sends a signal to the CPU, after the CPU detects the signal, it interrupts the current work and then processes the interruption. Then, the processor will notify the operating system that an interruption has occurred, so that the operating system will handle the interruption as appropriate. Now let's take a look at the interrupt controller. There are two common interrupt controllers: Programmable Interrupt Controller 8259A and advanced programmable interrupt controller APIC ), the Interrupt Controller should have been used in university courses on hardware interfaces and computer architecture. The traditional 8259A is only suitable for a single CPU. It is now a multi-CPU multi-core SMP system, intel introduces an advanced programmable interrupt controller (APIC) to make full use of the SMP architecture and transmit interruptions to each CPU on the system for better parallel implementation and performance improvement ).

Hardware Support for advanced programmable interrupt controllers is not enough, and the Linux kernel must be able to utilize these hardware features. Therefore, only Versions later than kernel 2.4 support IRQs for different hardware interrupt requests) allocated to a specific CPU. This binding technology is called smp irq Affinity. for more information, see the Linux kernel source code Documentation: linux-2.6.31.8/Documentation/IRQ-affinity.txt

How to Use

First, let's take a look at how the system's interruptions are distributed on the CPU. Obviously, there are more interruptions to be processed on the CPU 0:

# cat /proc/interrupts           CPU0       CPU1  0:  918926335          0    IO-APIC-edge  timer  1:          2          0    IO-APIC-edge  i8042  8:          0          0    IO-APIC-edge  rtc  9:          0          0   IO-APIC-level  acpi 12:          4          0    IO-APIC-edge  i8042 14:    8248017          0    IO-APIC-edge  ide0 50:        194          0   IO-APIC-level  ohci_hcd:usb2 58:      31673          0   IO-APIC-level  sata_nv 90:    1070374          0         PCI-MSI  eth0233:         10          0   IO-APIC-level  ehci_hcd:usb1NMI:       5077       2032LOC:  918809969  918809894ERR:          0MIS:          0

In order not to make CPU 0 tired, how can I transfer some interruptions to CPU 1? Or how to switch the interruption of the eth0 Nic to CPU1? First, check the smp affinity of the IRQ 90 interrupt to see how the current interrupt is allocated ffffff on different CPUs, which means it is allocated to all available CPUs ):

# cat /proc/irq/90/smp_affinity7fffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff

Before proceeding, we need to stop the service process automatically adjusted by IRQ so that we can manually bind IRQ to different CPUs, otherwise, the changes made by manually binding will be overwritten by the automatic adjustment process. To modify the interrupt handling of IRQ 90, bind it to 2nd cpucpu 1 ):

# /etc/init.d/irqbalance stop
# echo "2" > /proc/irq/90/smp_affinity

After a period of time, I watched/proc/interrupts. Is it 90: The eth0 interrupt on CPU1 increased by 145?) and the interrupt on CPU0 did not change? Print/proc/interrupts constantly and we will find that the number of eth0 interruptions on CPU0 remains unchanged, while the number of interruptions on CPU1 continues to increase. This is what we want:

# cat /proc/interrupts           CPU0       CPU1  0:  922506515          0    IO-APIC-edge  timer  1:          2          0    IO-APIC-edge  i8042  8:          0          0    IO-APIC-edge  rtc  9:          0          0   IO-APIC-level  acpi 12:          4          0    IO-APIC-edge  i8042 14:    8280147          0    IO-APIC-edge  ide0 50:        194          0   IO-APIC-level  ohci_hcd:usb2 58:      31907          0   IO-APIC-level  sata_nv 90:    1073399        145         PCI-MSI  eth0233:         10          0   IO-APIC-level  ehci_hcd:usb1NMI:       5093       2043LOC:  922389696  922389621ERR:          0MIS:          0

What is the purpose?

When the network is very heavy, for applications such as file servers and high-traffic Web servers, binding IRQ balances of different NICs to different CPUs will reduce the burden on a certain CPU, improves the overall capability of multiple CPUs to handle interruptions. For applications such as database servers, binding a disk controller to a CPU and a NIC to another CPU improves the database response time and performance. Balancing IRQ interruptions based on your production environment and application characteristics helps improve the overall throughput and performance of the system.

I often receive emails from netizens asking me how to optimize Linux, optimize VPS, and this is not a good answer. Remember that performance optimization is a process rather than a result, without reading some documents and modifying parameters, it is called optimization. A lot of tests, monitoring, and continuous observation and improvement are also required later.