Does the Linux TCP retransmission cause soft interrupts to be uneven across the cores?

One colleague, while troubleshooting the problem, said that because of the network instability, some of the core of the soft interrupt is very high. So, let's analyze the accuracy of this assertion.

Environment Description:

The NIC soft interrupt is tied to the core. The device has 80 cores, 960 network cards are interrupted, the BBR is not turned on, all TCP calls.

Cat /proc/cpuinfo | grep processor| WC -l

Cat /proc/interrupts | grep   ETH | WC -l960

cat /proc/irq/848/smp_affinity00000000,00000000,00000000, 00000000,00008000,00000000,00000000

Each NIC interrupt is specified on a CPU core.

Problem Description: Some nuclear soft interrupts were found to be much higher than other cores, because at that time there was a retransmission rate of about 2 points.

Analysis Process:

First of all, the retransmission of the number of messages, it is indeed possible to cause a soft interrupt increase, then we look at how the specific impact.

Tcp_v4_init_sock---->tcp_init_sock---->tcp_init_xmit_timers---->inet_csk_init_xmit_timers

voidInet_csk_init_xmit_timers (structSock *SK,void(*retransmit_handler) (unsignedLong),                   void(*delack_handler) (unsignedLong),                   void(*keepalive_handler) (unsignedLong)){    structInet_connection_sock *ICSK =INET_CSK (SK); Setup_timer (&icsk->Icsk_retransmit_timer, Retransmit_handler, (unsignedLong) SK); Setup_timer (&icsk->Icsk_delack_timer, Delack_handler, (unsignedLong) SK); Setup_timer (&sk->sk_timer, Keepalive_handler, (unsignedLong) SK); ICSK->icsk_pending = Icsk->icsk_ack.pending =0;}

We see that in this place set up the retransmission timer Icsk_retransmit_timer, according to an array of soft interrupts:

enum {    Hi_softirq=0,    Timer_softirq,    Net_tx_softirq,    Net_rx_softirq,    BLOCK _SOFTIRQ,    Block_iopoll_softirq,    Tasklet_softirq,    SCHED_SOFTIRQ,    /*  Unused, but kept as tools rely                on the*/    RCU_SOFTIRQ    ,/*   */    Nr_softirqs};

Our timer, is the use of TIMER_SOFTIRQ this interrupt number, then according to the interrupt number on the CPU distribution is uniform, you can see how much impact.

LOC:1072490396  678867892  808702249  738663051  694968438  664162833  654867572  651600329  654034656  645216969  627279751  639649619  641258373  639135342  635356673  641374876  637985751  629441830  632710723  631181922  860372920  811835568  721220382  669551672  649964241  632893235  628735585  624749341  622625858  616301049  619922385  612601767  612468721  614664270  616217297  614498541  610157489  607806218  610633457  604827668 1012416149  991506038  804318769  713451371  635498936  622276467  613574278  604650512  596876703  594943840  589362031  598699322  591033519  592127520  590210531  600537489  590585630  585989825  589738219  589555728  801802931  800267559  706742143  638665014  607582520  588593222  582048215  571846551  576418826  574034238  578342405  567753955  576671298  570976007  573807289  573966353  568759967  891048780  570053521  571002170Local Timer interrupts

Except for the cpu0, the other nuclei are very homogeneous.

In fact, the direct analysis of this problem/proc/interrupts can be seen, in fact, because some nuclear net_tx_softirq very little results.

So the problem comes, although we know the retransmission of the message to the CPU's soft interrupt number of unbalanced situation is very small, then why the call, the number of soft interrupts of each CPU is very large.

Back to the nature of the problem, our system actually turns on RPS and Rfs,rps (Receive Packet steering) primarily to balance the load of the soft interrupt to each CPU, simply, the NIC driver generates a hash ID for each stream, This hash is worth calculating by four tuples (Sip,sport,dip,dport). It looks like it's okay.

In fact, the problem is that we are multimedia server, the IP of the machine used for testing is not tampered with, our source IP and destination IP are fixed, the destination port is fixed when the first request is built, the subsequent flow is to rely on the source port and destination port to ensure hash hashing, by grasping packet discovery, In fact, this hash is not good. When we put the source IP configuration A lot of time, the phenomenon has been greatly improved, the purpose of the server to increase the IP more time, the effect is better.

Does the Linux TCP retransmission cause soft interrupts to be uneven across the cores?