Using SYSTEMTAP injection latency to simulate IO device jitter in Linux

This behavior can be tricky when our IO-intensive applications suspect device IO jitter, such as a period of long wait times leading to performance or other problems, because the hardware jitter is infrequent and costly to reproduce.

Luckily, Systemtap can save us. In principle, the IO we use is accessed through the file system, regardless of read/write/sync, and most of our files are opened in buffered mode. In this mode, if the Pagecache is not hit, you need to access the device. With this basic principle in view, we can achieve this by using a controlled injection delay in the universal Systemtap to the VFS's read and write requests.

The main points are as follows:
1. Controlled points of time.
2. Delay time controllable.
3. Target equipment controllable.

I wrote a script to inject IO latency, simulate SSD/FIO hardware jitter to verify that the IO jitter will affect the application, three steps as follows:
Step 1: Compile the module

The code is as follows

Copy Code

$ cat INJECT_KA.STP Global inject, ka_cnt Probe PROCFS ("CNT"). Read { $value = sprintf ("%dn", ka_cnt); } Probe PROCFS ("inject"). Write { Inject= $value; printf ("Inject count%d, ka%s", ka_cnt, inject); } Probe Vfs.read.return, Vfs.write.return { if ($return && Devname = @1 && inject = = "Onn") { ka_cnt++; Udelay ($); } } Probe begin{ println ("IK module begin:)"); } $ stap-v Systemtap translator/driver (version 2.1/0.152, commit release-2.0-385-gab733d5) Copyright (C) 2005-2013 Red Hat, Inc. and others This is free software; The source for copying conditions. Enabled Features:libsqlite3 NSS boost_shared_ptr tr1_unordered_map NLS $ sudo stap-p4-dmaxskipped=9999-m ik-g INJECT_KA.STP sda6 300 Ik.ko

Where the parameter sda6 is the name of the target device, 300 is the time to delay, the unit us (more than 300 is prone to error, because usually Systemtap will be the script to perform the CPU to check, too much CPU will trigger the protection mechanism, causing STAP complained of exit), is usually sufficient for SSD devices.

This step generates Ik.ko, verify that the machine that generates the module and the target machine, the operating system version is identical, and make sure that your STAP version is higher because the Udelay function is available in a higher version of STAP.

Step 2:

Copy the Ik.ko to the target machine and execute

The code is as follows	Copy Code
$ sudo staprun Ik.ko IK module begin:)

Step 3:
After starting the application to start the test, run the following command to start the injection:

The code is as follows	Copy Code
$ echo on|sudo tee/proc/systemtap/ik/inject && sleep && echo Off|sudo tee/proc/systemtap/ik/inject

Where Sleep N is the time when you want to open the injection switch.

Summary: Systemtap use good very invincible!

Have a good time!