One colleague, while troubleshooting the problem, said that because of the network instability, some of the core of the soft interrupt is very high. So, let's analyze the accuracy of this assertion.
Environment Description:
The NIC soft interrupt is tied to the core. The device has 80 cores, 960 network cards are interrupted, the BBR is not turned on, all TCP calls.
Cat /proc/cpuinfo | grep processor| WC -l
Cat /proc/interrupts | grep ETH | WC -l960
cat /proc/irq/848/smp_affinity00000000,00000000,00000000, 00000000,00008000,00000000,00000000
Each NIC interrupt is specified on a CPU core.
Problem Description: Some nuclear soft interrupts were found to be much higher than other cores, because at that time there was a retransmission rate of about 2 points.
Analysis Process:
First of all, the retransmission of the number of messages, it is indeed possible to cause a soft interrupt increase, then we look at how the specific impact.
Tcp_v4_init_sock---->tcp_init_sock---->tcp_init_xmit_timers---->inet_csk_init_xmit_timers
voidInet_csk_init_xmit_timers (structSock *SK,void(*retransmit_handler) (unsignedLong), void(*delack_handler) (unsignedLong), void(*keepalive_handler) (unsignedLong)){ structInet_connection_sock *ICSK =INET_CSK (SK); Setup_timer (&icsk->Icsk_retransmit_timer, Retransmit_handler, (unsignedLong) SK); Setup_timer (&icsk->Icsk_delack_timer, Delack_handler, (unsignedLong) SK); Setup_timer (&sk->sk_timer, Keepalive_handler, (unsignedLong) SK); ICSK->icsk_pending = Icsk->icsk_ack.pending =0;}
We see that in this place set up the retransmission timer Icsk_retransmit_timer, according to an array of soft interrupts:
enum { Hi_softirq=0, Timer_softirq, Net_tx_softirq, Net_rx_softirq, BLOCK _SOFTIRQ, Block_iopoll_softirq, Tasklet_softirq, SCHED_SOFTIRQ, /* Unused, but kept as tools rely on the*/ RCU_SOFTIRQ ,/* */ Nr_softirqs};
Our timer, is the use of TIMER_SOFTIRQ this interrupt number, then according to the interrupt number on the CPU distribution is uniform, you can see how much impact.
LOC:1072490396 678867892 808702249 738663051 694968438 664162833 654867572 651600329 654034656 645216969 627279751 639649619 641258373 639135342 635356673 641374876 637985751 629441830 632710723 631181922 860372920 811835568 721220382 669551672 649964241 632893235 628735585 624749341 622625858 616301049 619922385 612601767 612468721 614664270 616217297 614498541 610157489 607806218 610633457 604827668 1012416149 991506038 804318769 713451371 635498936 622276467 613574278 604650512 596876703 594943840 589362031 598699322 591033519 592127520 590210531 600537489 590585630 585989825 589738219 589555728 801802931 800267559 706742143 638665014 607582520 588593222 582048215 571846551 576418826 574034238 578342405 567753955 576671298 570976007 573807289 573966353 568759967 891048780 570053521 571002170Local Timer interrupts
Except for the cpu0, the other nuclei are very homogeneous.
In fact, the direct analysis of this problem/proc/interrupts can be seen, in fact, because some nuclear net_tx_softirq very little results.
So the problem comes, although we know the retransmission of the message to the CPU's soft interrupt number of unbalanced situation is very small, then why the call, the number of soft interrupts of each CPU is very large.
Back to the nature of the problem, our system actually turns on RPS and Rfs,rps (Receive Packet steering) primarily to balance the load of the soft interrupt to each CPU, simply, the NIC driver generates a hash ID for each stream, This hash is worth calculating by four tuples (Sip,sport,dip,dport). It looks like it's okay.
In fact, the problem is that we are multimedia server, the IP of the machine used for testing is not tampered with, our source IP and destination IP are fixed, the destination port is fixed when the first request is built, the subsequent flow is to rely on the source port and destination port to ensure hash hashing, by grasping packet discovery, In fact, this hash is not good. When we put the source IP configuration A lot of time, the phenomenon has been greatly improved, the purpose of the server to increase the IP more time, the effect is better.
Does the Linux TCP retransmission cause soft interrupts to be uneven across the cores?