Linux Server advanced: Who is switching our processes?

When running a Linux server, we often need to know who is performing process switching and what causes process switching. Because process switching costs a lot, I will give a number from the LMbench test:

Context switching – times in microseconds – smaller is better————————————————————————-Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64Kctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw——— ————- —— —— —— —— —— ——- ——-my174.cm4 Linux 2.6.18- 6.1100 7.0200 6.1100 8.7400 7.7200 8.96000 9.62000

On my high-end server, the overhead of process switching is about 8 us, which is unacceptable to high-performance servers. Therefore, we need to do as much as possible in a time slice, instead of wasting time on unnecessary switching.

Let's find out who is switching our process:

[root@my174 admin]# dstat 1—-total-cpu-usage—- -dsk/total- -net/total- —paging– —system–usr sys idl wai hiq siq| read writ| recv send| in out | int csw0 0 100 0 0 0| 0 0 | 796B 1488B| 0 0 |1004 1280 0 100 0 0 0| 0 0 | 280B 728B| 0 0 |1005 1140 0 100 0 0 0| 0 0 | 280B 728B| 0 0 |1005 1280 0 100 0 0 0| 0 0 | 280B 728B| 0 0 |1005 1140 0 100 0 0 0| 0 320k| 280B 728B| 0 0 |1008 143…

We can see that the number of CSWs is 120/S, but tools similar to dstat or vmstat do not tell us who is doing bad things. Okay! Let's do it ourselves.

Offering our lovely systemtap!

[root@my174 admin]# cat >cswmon.stp#! /usr/bin/env stap## global csw_countglobal idle_count probe scheduler.cpu_off {csw_count[task_prev, task_next]++idle_count+=idle} function fmt_task(task_prev, task_next){return sprintf(“%s(%d)->%s(%d)”,task_execname(task_prev),task_pid(task_prev),task_execname(task_next),task_pid(task_next))} function print_cswtop () {printf (“%45s %10s\n”, “Context switch”, “COUNT”)foreach ([task_prev, task_next] in csw_count- limit 20) {printf(“%45s %10d\n”, fmt_task(task_prev, task_next), csw_count[task_prev, task_next])}printf(“%45s %10d\n”, “idle”, idle_count) delete csw_countdelete idle_count} probe timer.s($1) {print_cswtop ()printf(“————————————————————–\n”)}CTRL+D

This script will print the TOP 20 most switched processes and their pid at the specified time. Let's take a look at the result:

[root@my174 admin]# stap cswmon.stp 5Context switch COUNTswapper(0)->systemtap/11(908) 500systemtap/11(908)->swapper(0) 498swapper(0)->fct1-worker(2492) 50fct1-worker(2492)->swapper(0) 50swapper(0)->fct0-worker(2191) 50fct0-worker(2191)->swapper(0) 50swapper(0)->bond0(3432) 50bond0(3432)->swapper(0) 50stapio(879)->swapper(0) 26swapper(0)->stapio(879) 25stapio(879)->swapper(0) 19swapper(0)->stapio(879) 17swapper(0)->watchdog/9(31) 5watchdog/9(31)->swapper(0) 5swapper(0)->mysqld(18346) 5mysqld(18346)->swapper(0) 5swapper(0)->watchdog/13(43) 5watchdog/13(43)->swapper(0) 5swapper(0)->watchdog/14(46) 5watchdog/14(46)->swapper(0) 5idle 859————————————————————–…

We can see where the process is switched and how many times it happened. In the last line, I printed the number of idle times. That is to say, when the system has nothing to do, It switches to idle (0) this process goes to rest.

Through the above investigation, we will clearly understand the overhead of our system, so that we can locate the problem easily.

Have fun!