The Tasklet scheduling problem of netback and the simple analysis of network card packet loss

Recently on the million Gigabit network card test, there has been a phenomenon that the Gigabit network card did not appear, Tasklet version of the netback, the VM to carry out the package test, found that vif interrupt default binding on CPU0, but the VM packet run found on the host CPU1, The ksoftirqd of CPU2 is very high.

From the previous understanding, the package from NetFront out through the Eventchannel notify trigger the Vif IRQ handler function, and then tasklet_schedule call Tx_action, through a series of processing process to send packets to the network card. So vif interrupt tied to which CPU, when the contract, the CPU si% well paid pair.

A closer look at Tasklet's code found that, in addition to the interrupt handler function called Tasklet_schedule, there is a place to call, that is, Netif_idx_release, which is a callback function to use with Grant map. When the SKB from the network card issued, KFREE_SKB time, will call the page registration of a callback, tell Netback, this SKB send completed, you can send NetFront, and then response.

So from here, it is not difficult to see, CPU1, Cpu2 on the Softirq high, Tasklet is also high, because the network card interrupts are in Cpu1, Cpu2.

At first I was a little bit silly, think TX, how can trigger the network card interrupt, should not be RX when there is. Later saw the IGB drive, only to find really silly.

Network card after the package, you have to SKB recycling ah, this time, is by triggering the REQUEST_IRQ function before the completion of the package is to go this function.

Take the line tens of thousands of gigabit network card Ixgbe

Interrupt handler function is Ixgbe_msix_clean_many, inside call Napi_schedule, go to Ixgbe_poll. This function is the one we mentioned above to handle the package.

It calls IXGBE_CLEAN_TX_IRQ first, recycles the packets sent by the previous TX (Ixgbe_unmap_and_free_tx_resource), and then calls IXGBE_CLEAN_RX_IRQ

This inside will rx_ring inside the rx_buffer_info including the SKB, call the stack of the packet function, throw to the upper processing, ixgbe_receive_skb->napi_gro_receive->netif_receive _skb

This rx_ring count in the NIC driver load by default is 1024, we set the maximum limit of 4,096, is the driver inside each receive a batch of packets, it will ixgbe_alloc_rx_buffers assign a batch of new SKB and page

PCI_DMA mapping to the hardware to collect packets

Here the first Tasklet dispatch to CPU1, cpu2 the problem can be explained, the VM packet triggered the network card cpu1, cpu2 on the interruption, until soft interrupt, KFREE_SKB, Trigger Idx_release, then tasklet_schedule, then TX _action has been on these two CPUs, and then vif the contract to trigger the interrupt, call Maybe_schedule_tx_action, inside will determine whether the current pending is less than MAX/2, if less than to call tasklet_ Schedule, so even if it is called, the Tasklet may already be running. Why before the gigabit network did not appear, it may be because the idx_release is called to become faster, not to confirm, it is not important.

There is also a network card packet loss problem, with the NIC driver a bit of a relationship, a test found that the physical mouth received a 14w packet, lost 45w package, ethtool-s See is rx_fifo_errors, this probably indicates that because there is no buffer caused. Just above also talked about Rx_buffer, is after processing a batch of requests to allocate a new batch of buffer, a total of 4,096. If CPU processing slows down, there will be a lot of drops in the case of big pressure outside, which is related to CPU processing power.