Who is switching our processes in Linux?

When running a Linux server, we often need to know who is performing process switching and what causes process switching. Because process switching costs a lot, I will give a number from the lmbench test:
Context switching-times in microseconds-smaller is better
-------------------------
Host OS 2 P/0 K 2 P/16 K 2 P/64 K 8 p/16 K 8 p/64 K 16 p/16 K 16 p/64 K
Ctxsw
------------------------
My174.cm4 Linux 2.6.18-6.1100 7.0200 6.1100 8.7400 7.7200 8.96000

On my high-end server, the overhead of process switching is about 8 us, which is unacceptable to high-performance servers. Therefore, we need to do as much as possible in a time slice, instead of wasting time on unnecessary switching.

Let's find out who is switching our process:

View Source

Print?

[root@my174 admin]# dstat 1

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| readwrit| recv  send|  inout | int   csw

0   0 100   0   0   0|   0     0 | 796B 1488B|   0     0 |1004   128

0   0 100   0   0   0|   0     0 | 280B  728B|   0     0 |1005   114

0   0 100   0   0   0|   0     0 | 280B  728B|   0     0 |1005   128

0   0 100   0   0   0|   0     0 | 280B  728B|   0     0 |1005   114

0   0 100   0   0   0|   0   320k| 280B  728B|   0     0 |1008   143

...

We can see that the number of CSWs is 120/s, but tools similar to dstat or vmstat do not tell us who is doing bad things. Okay! Let's do it ourselves.
Offering our lovely systemtap!

View Source

Print?

[root@my174 admin]# cat >cswmon.stp

#! /usr/bin/env stap

#

#

global csw_count

global idle_count

probe scheduler.cpu_off {

csw_count[task_prev, task_next]++

idle_count+=idle

}

function fmt_task(task_prev, task_next)

{

return sprintf("%s(%d)->%s(%d)",

task_execname(task_prev),

task_pid(task_prev),

task_execname(task_next),

task_pid(task_next))

}

function print_cswtop () {

printf ("%45s %10s/n", "Context switch", "COUNT")

foreach ([task_prev, task_next] in csw_count- limit 20) {

printf("%45s %10d/n", fmt_task(task_prev, task_next), csw_count[task_prev, task_next])

}

printf("%45s %10d/n", "idle", idle_count)

delete csw_count

delete idle_count

}

probe timer.s($1) {

print_cswtop ()

printf("--------------------------------------------------------------/n")

}

CTRL+D

This script will print the top 20 most switched processes and their PID at the specified time. Let's take a look at the result:

View Source

Print?

[root@my174 admin]# stap cswmon.stp 5

Context switch      COUNT

swapper(0)->systemtap/11(908)        500

systemtap/11(908)->swapper(0)        498

swapper(0)->fct1-worker(2492)         50

fct1-worker(2492)->swapper(0)         50

swapper(0)->fct0-worker(2191)         50

fct0-worker(2191)->swapper(0)         50

swapper(0)->bond0(3432)         50

bond0(3432)->swapper(0)         50

stapio(879)->swapper(0)         26

swapper(0)->stapio(879)         25

stapio(879)->swapper(0)         19

swapper(0)->stapio(879)         17

swapper(0)->watchdog/9(31)          5

watchdog/9(31)->swapper(0)          5

swapper(0)->mysqld(18346)          5

mysqld(18346)->swapper(0)          5

swapper(0)->watchdog/13(43)          5

watchdog/13(43)->swapper(0)          5

swapper(0)->watchdog/14(46)          5

watchdog/14(46)->swapper(0)          5

idle        859

--------------------------------------------------------------

...

We can see where the process is switched and how many times it happened. In the last line, I printed the number of idle times. That is to say, when the system has nothing to do, It switches to idle (0) this process goes to rest.

Through the above investigation, we will clearly understand the overhead of our system, so that we can locate the problem easily.
Have fun!

 

Reprinted: http://blog.yufeng.info/archives/tag/csw