Frequent switching of process contexts in Linux causes load average to be too high

First, the phenomenon of problems

Now network has two virtual machine host 95% CPU in idle state, memory usage is not particularly high, and the host load average reached more than 40.

Second, the problem analysis

First on the host through the top, free, PS, iostat and other common tools to analyze the host CPU, memory, IO usage, found that the three are not high. The results viewed through Vmstat 1 are as follows:

From the output of the Vmstat, the block in and block out of IO items are not frequent. The system item's per-number of interrupts (in), context switches per second (CS) are particularly frequent. This causes the load Avaerage to be particularly high. The general direction of the root because found, specifically which process how frequently interrupt and the file switch?

Here we use pidstat-w 1 (Refresh output context switch per second), the output is shown in the following figure:

From the above, you can see that there are CSWCH (voluntary context switching) and NVCSWCH (involuntary context Switching) and corresponding commands, vsftpd occupy a lot of file exchange. Can see here the CS values and total value there is a relatively large gap, because the host started more than one vsftpd process, and pidstat through 1 seconds refresh will not show all, through the Pidstat- W performs several collections of all found VSFTPD processes occupied by the CS value Overlay and Vmstat in the comparison.

When the result is communicated to the business person, and business people's guess is also consistent, because FTP use of the directory structure of a deeper level, the number of files are also more, the business in the backup of the old directory and recreate a single layer of directory, after observing a paragraph, found that the load average lowered, stable under 1.

Of course, here is just a kind of processing methods, the current network some programs do not make such a modification, and do not let the process between the CPU frequently switch, but also by setting the fixed running CPU on the tuning method, the following two processes running on the 0-7 CPU:

[Root@www ~]# taskset-c-P 6389
PID 6389 ' s current affinity list:0-7
[Root@www ~]# taskset-c-P 6580
PID 6580 ' s current affinity list:0-7
It can be fixed on the number 0-1 CPU by Taskset:

[Root@www ~]# taskset-c 0,1-p 6389
The rationale for this is that each time the process switches to the next CPU core to flush the current cache data, specifying the CPU reduces such operations and increases the processing speed of the process. The old program is more effective when it is tuned.

Iii. related Context Switching

1, the understanding of context switching

What is up and down file switching? Quote foreigner: A Context switch (also sometimes referred to as A process switch or a task switch) is the switching of the CPU (c Entral processing unit) from one process or thread to another. For more information, refer to the Linfo site or Wikipedia.

Too much of the context switch can cause the CPU to act like a porter, frequently rushing between registers and running queues, more time spent on thread switching than on threads that actually work. Direct consumption includes CPU registers that need to be saved and loaded, and the system Scheduler code needs to be executed. Indirect consumption is shared data between multiple-core cache.

2, cause of context switching

For a preemptive operating system, there are generally several:

After the current task's time slice is used up, the system CPU normally dispatches the next task;
The current task encounters IO blocking, and the dispatch thread hangs the task and continues on to the next task;
Multiple tasks preemption lock resources, the current task did not grab, the scheduler hangs, continue the next task;
The user code hangs the current task, giving up the CPU time;
hardware interruption;
What kind of operation will cause CS, here is a blog post feel very good, although the part of the code is not understood. There are a few words:

When the time slice of a process in Linux expires, or when a process preemption with a higher priority occurs, CS will occur, but these are not controllable by our application developers--The previous section is well described, and the latter part of the system level and kernel development level can call nice or Renice to set priority to ensure that some programs take precedence over the CPU time, but also can not be refined to the CS level.

From the developer's perspective, our process can actively apply to the kernel for CS. To do this: Hibernate the current process/thread, and wake up other processes/threads.

3. Context Switch Test tool

1, Lmbench is the bandwidth (read cache files, memory copy, read and write memory, pipelines, etc.) and reaction time (context switching, network, process creation, etc.) evaluation tool;

2, Micro-benchmark Contextswitch can test different CPUs at least how many NS can be a file switch, and then converted to seconds, we can confirm that the processor each can be the upper and lower file switching number, the tool can be used to see the Tsuna blog.

4, the context switch view method

Sar-w, this just shows the total upper and lower file switching on the host.

# sar-w 1
proc/s
Total number of tasks created/second.
cswch/s
Total number of context switches/second.
Similarly, Vmstat can view the overall context switch, but the Vmstart output results more than a comparison to find the problem:

# Vmstat 3
procs-----------Memory-------------Swap-------IO-----system------CPU----
R b swpd free buff cache si so bi bo in CS US sy ID WA
2 0 7292 249472 82340 2291972 0 0 0 0 0 0 7 13-79 0
0 0 7292 251808 82344 2291968 0 0 0 184 24 20090 1 1-99 0
0 0 7292 251876 82344 2291968 0 0 0 83 17 20157 1 0-99 0
0 0 7292 251876 82344 2291968 0 0 0 73 12 20116 1 0-99 0
View the use of the upper and lower files for each process or thread, either by using the Pidstat command or by viewing the Proc.

# PIDSTAT-W The context switching of each process
# PIDSTAT-WT subdivided into each threads
View the file methods under proc as follows:
# pid=307
# grep ctxt/proc/$pid/status
voluntary_ctxt_switches:41 #自愿的上下文切换
Nonvoluntary_ctxt_switches:16 #非自愿的上下文切换
CSWCH/S: The number of active (voluntary) switching contexts per second, and when a task is blocked waiting, it actively yields its own CPU resources.

NVCSWCH/S: The number of passive (involuntary) transitions for a task per second, the time slice that the CPU has allocated to a task has run out, and will force the process to relinquish the CPU's execution power.

Context switching part of bits and pieces first come here, just want to explain context switching is a more important indicator. Nagios CHECK_MK default to the context of the monitoring, the method is to use the two/proc/stat file to the Ctxt row, and take two time between the difference between the value to confirm.

# Cat/proc/stat|grep Ctxt
Ctxt 111751207

S