Solution to the performance bottleneck caused by the soft interruption of the ksoftirqd process consuming a single core 100% si Processing

Linux Virtual Server (LVS): ksoftirqd processes run out of Single-core 100% si Processing Soft interruptions, leading to performance bottlenecks

Recently tested LVS performance, and found that the system reaches its peak performance when one of the CPU cores is exhausted.

Ksoftirqd processes consume CPU resources, all of which are used to handle soft interruptions (the process name can also be identified ).

After searching, many people have encountered such problems and it seems that they have not solved them. The following solutions are available:

1. Reduce the number of cluster members;

2. Modify the cluster mode (NAT, TURNL, and DR );

3. Modify the cluster scheduling algorithm;

4. Upgrade the operating system kernel to 2.6.20 or later;

5. Adjust the maximum transmission unit (MTU) of the NIC );

6. Modify the device interruption mode;

7. Use multi-nic Server Load balancer;

8. Upgrade the hardware (NIC );

9. Change the operating system.

Let's explain it one by one:

1st: Reduce the number of cluster members. Because the bottleneck is not on the Real Server, The lvs performance does not change significantly when the number of members is reduced.

2nd: Modify the cluster mode. Theoretically, the DR model is the most resource-saving, and should be known to anyone who knows about the theory. Because the NAT mode does not meet the requirements, only the DR and TUN modes are compared. There is no significant difference between the two modes.

3rd: Modify the cluster scheduling algorithm. Among the ten existing algorithms, rr is the simplest, And the bottleneck has not yet reached this level. Actually, the bottleneck caused by processing network packets. The scheduling algorithm simply compares rr and wrr, and there is no obvious difference between them.

4th: Upgrade the operating system kernel to 2.6.20 or later. I directly upgraded to the latest version 2.6.34, but the result bottleneck has not been improved.

5th: Adjust the maximum transmission unit of the network card. The maximum transmission unit supported by the vswitch is 9216, and the maximum transmission unit of the NIC is changed to 1500 (default), 5000, 9000, and 9216 respectively. Among them, there is no significant difference between 1500 and 5000. 9000 and 9216 may cause network instability and performance does not improve, but a large number of connection times out.

6th: Modify the device interruption mode. Modify the settings to interrupt/proc/irq/$ {Nic interrupt number}/smp_affinity:

The CPU of the test server is quad-core. The smp_affinity value of the theoretical network adapter is 1, 2, 4, and 8, which correspond to cpu0, cpu1, cpu2, and cpu3 respectively.

Result:

1. The default smp_affinity value of the NIC is 8. During the test, all soft interruptions are handled by cpu3. Correct

2. Set smp_affinity = 1. During the test, all soft interruptions are handled by cpu0. Correct

3. Set smp_affinity to 2. During the test, all soft interruptions are handled by cpu1. Correct

4. Set smp_affinity = 4. During the test, all soft interruptions are handled by cpu2. Correct

5. Set smp_affinity = 5. During the test, all soft interruptions are handled by cpu0, which should be allocated to cpu0 and cpu2. Invalid

6. Set smp_affinity = f. During the test, all soft interruptions are handled by cpu0, which should be allocated to cpu0, cpu1, cpu2, and cpu2. Invalid

That is, the smp_affinity modification function is only valid for Single-core applications.

7th: use multi-nic Server Load balancer. This solution is feasible! When two NICs are bound to one IP address, the performance is doubled, and the effect is very obvious. The reason is that the two NICs each use one CPU core, which naturally doubles the performance compared with the single core.

The configuration method is as follows:

Single Nic Working Mode

# Cat/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE = eth0
BOOTPROTO = none
BROADCAST = 192.168.223.255
HWADDR = 00: 1E: 90: 76: 6F: E0
IPADDR = 192.168.223.113
NETMASK = 255.255.254.0
NETWORK = 10.20.222.0
ONBOOT = yes
GATEWAY = 192.168.222.1
TYPE = Ethernet

Procedure

Echo 'Alias bond0 bonding '>/etc/modprobe. conf

# Cat/etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE = bond0
BOOTPROTO = static
BROADCAST = 192.168.223.255
MACDDR = 00: 1E: 90: 76: 6F: E2
IPADDR = 192.168.223.113
NETMASK = 255.255.254.0
NETWORK = 192.168.222.0
USERCTL = no
ONBOOT = yes
GATEWAY = 10.20.222.1
TYPE = Ethernet
BONDING_OPTS = "mode = 0 miimon = 100"

# Cat/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE = eth0
ERCTL = no
ONBOOT = yes
MASTER = bond0
SLAVE = yes
BOOTPROTO = none

# Cat/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE = eth1
USERCTL = no
ONBOOT = yes
MASTER = bond0
SLAVE = yes
BOOTPROTO = none

# Service network restart

8th: Upgrade the hardware to use a NIC that supports RSS.

Below is intel's description of RSS

Receive-side scaling (RSS) routes incoming packets to specific queues, efficiently balancing network loads memory ss cpu cores and increasing performance on multi-processor systems. RSS, called Scalable I/O in Linux *, creates a hash table from IP, TCP, and Port Addresses and uses that table to decide which queue to route a packet, and to which processor the packet shocould be associated.

However, RSS is supported from the hardware metrics on the official website where we use NICs. For Windows, set 'netsh int tcp set global rss = enabled '.

Change the operating system. This solution is less likely to be deployed in the production environment, but it is necessary to confirm whether the operating system is off.

It is said that NLB, solaris, And AIX of Windows support Nic RSS, which remains to be confirmed