The latest I/O statistics Tool in CentOS

The latest I/O statistics Tool in CentOS

As a Linux system administrator, it is essential to count various types of IO. Iostat is obviously the most important statistical tool. However, iostat is not the focus of this article, because there are already a lot of tutorials on the network for your reference. Here, I want to introduce some other statistical tools to meet different requirements.

Iostat

Iostat is exceptionally powerful and has many output items, as shown in the following example:

123

Device: rrqm/s  wrqm/s  r/s     w/s    rkB/s    wkB/s    avgrq-sz avgqu-sz   await r_await w_await  svctm  %utilsda     0.00     0.50  173.50   73.50  3076.00   604.00    29.80   149.93    676.58   74.36 2098.15  4.05 100.00

Their meanings are as follows:

• Rrqm/s: the number of merge read operations per second. That is, delta (rmerge)/s.
• Wrqm/s: Number of write operations performed on merge per second. That is, delta (wmerge)/s
• R/s: The number of read I/O devices completed per second. That is, delta (rio)/s.
• W/s: the number of write I/O devices completed per second. That is, delta (wio)/s
• Rsec/s: Number of read sectors per second. That is, delta (rsect)/s
• Wsec/s: Number of write sectors per second. That is, delta (wsect)/s
• RkB/s: the number of K Bytes read per second. It is half of rsect/s, because the size of each sector is 512 bytes. (calculated)
• WkB/s: the number of K bytes written per second. Half of wsect/s (to be calculated)
• Avgrq-sz: average data size (slice) of each device I/O operation. delta (rsect + wsect)/delta (rio + wio)
• Avgqu-sz: Average I/O queue length, that is, delta (aveq)/s/1000 (because aveq is measured in milliseconds ).
• Await: average wait time for each device I/O operation (MS). That is, delta (ruse + wuse)/delta (rio + wio)
• Svctm: Average service time for each device I/O operation (MS). That is, delta (use)/delta (rio + wio)
• % Util: the percentage of time in one second for I/O operations, or the number of I/O queues in one second is not empty. that is, delta (use)/s/1000 (because the Unit of use is milliseconds)

If%utilClose to 100%, indicating that too many I/O requests are generated and the I/O system is fully loaded. This disk may have a bottleneck.

When the idle is less than 70% I/O, the load is high. Generally, the read speed is wait.

You can also combinevmstatView parameters B (number of processes waiting for resources) and wa (percentage of CPU time occupied by IO wait, higher than 30% when the IO pressure is high)

In additionawaitMore parameters are required.svctmFor reference. IO problems may occur if the difference is too high.

avgrq-szIt is also a point to note when performing IO optimization. This is the data size of each operation. If the number of times is large, but the data size is small, the IO will be very small. if the data is big, the IO data will be high. you can also useavgqu-sz × ( r/s or w/s ) = rsec/s or wsec/sThat is to say, the speed of reading is determined by this.

svctmGenerally lessawait(Because the wait time of the simultaneously waiting request is calculated repeatedly ),svctmThe size is generally related to disk performance, and the CPU/memory load will also affect it, and too many requests will indirectly causesvctm.awaitDepends on the service time (svctm) And the length of the I/O queue and the sending mode of the I/O Request. if svctm is close to await, it means that I/O has almost no waiting time. If await is much larger than svctm, it means that the I/O queue is too long and the response time of the application is slow, if the response time exceeds the user's allowable range, you can consider replacing a faster disk and adjusting the kernel.elevatorAlgorithm, optimize the application, or upgrade the CPU.

Queue Length (avgqu-sz) Can also be used as an indicator to measure the system I/O load, but becauseavgqu-szIs based on the average per unit time, so it cannot reflect the instantaneous I/O flood.

If you have time, I will write several separate posts to explainiostat.

Iodump

Iodump is a disk I/O tool used to count the consumption of every process (thread. This perl script, in principle, enables the kernel logging message switch for I/O, then reads the message and analyzes the output. The procedure is as follows:

First download this tool

wget http://aspersa.googlecode.com/svn/trunk/iodump

Enable the I/O kernel message

echo 1 >/proc/sys/vm/block_dump

After the above switch is enabled, the kernel will record the messages of each I/O operation. We only need to obtain and analyze them on a regular basis, such as the following:

while true; do sleep 1; dmesg -c ; done |perl iodump

Wait for a while and then passctrl+cTo end the above script, you will get the following similar information:

1234567

TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICESpostgres              5799       1919       1919          0          0 sda7jbd2/sda7-8           1572         35          0         35          0 sda7jbd2/sda2-8            250         32          0         32          0 sda2flush-8:0             2229         31          0         31          0 sda2, sda7postgres              4308          2          0          2          0 sda7bash                  5804          1          0          1          0 sda2

The output unit is block. The size of each block depends on the block size specified during file system creation. For example, the size of the block in my sda7 is 1 kb.

Iotop

Iotop is a tool written in Python, similartopTool UI, including some parameters andtopSimilar. However, it has some requirements for the system:

1. Python ≥ 2.5 or Python ≥ 2.4 with the ctypes module
2. Kernel ≥ 2.6.20
3. Kernel uses options:
   1. TASK_DELAY_ACCT
   2. CONFIG_TASKSTATS
   3. TASK_IO_ACCOUNTING
   4. CONFIG_VM_EVENT_COUNTERS

For RPM-based systems, you can directly download the compiled Binary Package (here) or binary source code package (here)

For Debian/Ubuntu systems, use

sudo apt-get install iotop

You can (I have to say that the software provided by the Debian system is quite rich). Other systems can download the source code through the following commands, and then compile

git clone git://repo.or.cz/iotop.git

For details about how to use iotop (8), refer to the iotop (8) Manual. The following is a display on my machine:

123456

iotop -o -u wgzhaoTotal DISK READ:       2.15 M/s | Total DISK WRITE:    1601.15 K/s  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND 5984 be/4 wgzhao      2.15 M/s   70.55 K/s  0.00 % 83.67 % postgres: wgzhao pgbench [local] UPDATE 4305 be/4 wgzhao      0.00 B/s  227.34 K/s  0.00 %  0.00 % postgres: writer process 4308 be/4 wgzhao      0.00 B/s   90.15 K/s  0.00 %  0.00 % postgres: stats collector process

Iopp

Iopp is another tool used to count the I/O of every process. It is written in C language and theoretically it is more efficient than the preceding two methods. The installation method is very simple. First download the source code through the following command:

git://github.com/markwkm/iopp.git

Then compile and install the SDK separately using the following commands:

123

cmake CMakeLists.txtmakemake install DESTDIR=/usr

The following is an example

1234567891011121314

iopp -i -c 2  pid    rchar    wchar    syscr    syscw   rbytes   wbytes  cwbytes command 2144        0      296       40        8        0        0        0 /usr/sbin/LCDd 2284        0        0        2        0        0        0        0 ha_logd: read process 2299        0        0        2        0        0        0        0 ha_logd: write process 2520        3        3        3        3        0        0        0 /usr/lib/virtualbox/vboxwebsrv 2599        2        2        2        2        0        0        0 /usr/lib/virtualbox/VBoxSVC 2675        0        0        1        0        0        0        0 runsvdir 3177       16       16        4        2        0        0        0 /usr/bin/gnome-shell 3192       16       16        4        2        0        0        0 nautilus 3305      180      340      100       60        0        0        0 /usr/lib/icedove/icedove-bin 3623     1393     1440        1        1        0        0        0 sshd: wgzhao@pts/0 4305        0  4603904        0      562        0  4603904        0 postgres: writer process    6257  2064384  1892352      252      215  3719168   139264        0 postgres: wgzhao pgbench [local] UPDATE

The meanings of the above output are as follows:

• Pid process ID
• The number of bytes that rchar will read from the disk
• Number of bytes that wchar has written or should be written to the disk
• Syscr read I/O count
• Syscw write I/O count
• The number of bytes that rbytes actually reads from the disk
• Number of bytes that wbytes actually writes to the disk
• The number of bytes that are not operated by cwbytes because the page cache is cleared
• Command

Whererbytes,wbytes,cwbytesWill be given-kOr-mAnd is displayedrkb,wkb,cwkbOrrmb,wmb,cwmb.commandIf one column is given-cThe complete command name is displayed, not just the command itself. Use andtopSimilar.

For more details, refer to the iopp (8) manual.

Dstat

Dstat is known as a variety of resource statistics tools, and its purpose is to replacevmstat,iostat,netstat,ifstatAnd other single statistical tools to achieveAll in one. Dstat is written in Python.

Dstat can clearly display the information of each column, especially the unit and size. It does not confuse or make mistakes in unit conversion. Most importantly, because it is based on modular design, we can easily write a plug-in to collect the statistics we need.

In addition, dstat output can also be exportedCSVFormat file, which allows you to easily generate statistical images in the workbook tool.

Currently, many dstat plug-ins are available. This is the current output on my machine:

123456789101112

$ dstat  --listinternal:  aio, cpu, cpu24, disk, disk24, disk24old, epoch, fs, int, int24, io, ipc, load, lock, mem, net,   page, page24, proc, raw, socket, swap, swapold, sys, tcp, time, udp, unix, vm/usr/share/dstat:  battery, battery-remain, cpufreq, dbus, disk-tps, disk-util, dstat, dstat-cpu, dstat-ctxt,   dstat-mem, fan, freespace, gpfs, gpfs-ops, helloworld, innodb-buffer, innodb-io, innodb-ops, lustre,   memcache-hits, mysql-io, mysql-keys, mysql5-cmds, mysql5-io, mysql5-keys, net-packets, nfs3,   nfs3-ops, nfsd3, nfsd3-ops, ntp, postfix, power, proc-count, qmail, rpc, rpcd, sendmail, snooze,   squid, test, thermal, top-bio, top-bio-adv, top-childwait, top-cpu, top-cpu-adv, top-cputime,   top-cputime-avg, top-int, top-io, top-io-adv, top-latency, top-latency-avg, top-mem, top-oom, utmp,   vm-memctl, vmk-hba, vmk-int, vmk-nic, vz-cpu, vz-io, vz-ubc, wifi

Below are a few used columns (the actual output is color-coded and can be easily identified)

Default dstat output

1234567891011121314151617181920

wgzhao-nb:~# dstatYou did not select any stats, using -cdngy by default.----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw   2   1  87  10   0   0| 816k  385k|   0     0 |   0     0 |2279  7048   5   1  78  16   0   0|2600k    0 | 140B  940B|   0     0 |5952    13k  5   3  80  12   0   0|2896k  182k|  70B  358B|   0     0 |6074    14k  4   2  78  16   0   0|2724k    0 |  70B  374B|   0     0 |5703    15k  4   2  81  14   0   0|3008k    0 |  70B  358B|   0     0 |5924    13k  5   1  80  14   0   0|1976k   17k|  70B  358B|   0     0 |5819    13k  5   2  79  14   0   0|2056k    0 | 198B  374B|   0     0 |5618    13k  4   2  79  15   0   0|2416k    0 |  70B  358B|   0     0 |5866    15k  5   2  78  15   0   0|2528k    0 |  70B  358B|   0     0 |6356    14k  5   2  78  16   0   0|2288k    0 |  70B  358B|   0     0 |6515    15k  5   2  79  14   0   0|2656k 8192B|  70B  358B|   0     0 |6490    15k  3   2  81  13   0   0|2296k    0 |  70B  374B|   0     0 |5573    15k  4   3  76  17   0   1|2224k    0 |  70B  358B|   0     0 |5366    12k  5   1  81  13   0   0|2208k    0 | 508B  358B|   0     0 |5403    13k  4   2  79  15   0   0|2024k  182k|  70B  358B|   0     0 |5583    13k  5   2  79  15   0   0|2148k   17k| 186B  490B|   0     0 |5400    12k

Columns to be displayed

12345678910111213141516

wgzhao-nb:~# dstat  -c --top-cpu -d --top-bio --top-latencyModule dstat_top_latency failed to load. (Kernel has no scheduler statistics, use at least 2.6.12)----total-cpu-usage---- -most-expensive- -dsk/total- ----most-expensive----usr sys idl wai hiq siq|  cpu process   | read  writ|  block i/o process     2   1  87  10   0   0|gnome-shell  0.7| 826k  384k|postgres    692k   52k  4   2  79  16   0   0|postgres: wgz3.0|1744k  776k|postgres: w1744k   72k  5   3  78  15   0   0|postgres: wgz5.0|3120k    0 |postgres: w3064k  136k  6   2  73  19   0   0|postgres: wgz4.2|2608k  285k|postgres: w2608k  136k  4   2  77  17   0   0|postgres: wgz3.5|2112k  848k|postgres: w2112k   88k  3   2  71  25   0   0|postgres: wgz2.0| 944k 1049k|postgres: w 936k   48k  3   2  58  37   0   0|postgres: wgz2.0| 920k 2070k|postgres: w 928k   64k  3   2  62  34   0   0|postgres: wgz2.2|1496k  992k|postgres: w1608k   72k  3   2  56  38   0   0|postgres: wgz3.0|1840k  645k|postgres: w1856k   88k  3   2  78  17   0   0|postgres: wgz3.0|1420k 1200k|postgres: w1292k   80k  5   2  80  12   0   1|postgres: wgz4.2|2628k    0 |postgres: w2636k  112k  4   3  69  25   0   0|postgres: wgz3.8|2168k  576k|postgres: w2224k  104k

Specify the columns to be displayed and export the results to the file.

123456789101112131415

wgzhao-nb:~# dstat  --disk --mem --proc --io --sys --filesystem --tcp --vm --output dstat.csv-dsk/total- ------memory-usage----- ---procs--- --io/total- ---system-- --filesystem- ----tcp-sockets---- -----virtual-memory---- read  writ| used  buff  cach  free|run blk new| read  writ| int   csw |files  inodes|lis act syn tim clo|majpf minpf alloc  free 844k  404k| 829M 19.4M 2920M  124M|  0 0.0 0.7|47.5  38.4 |2336  7185 | 4928  12286 | 11   3   0   0   2|   1   620   602   605 2128k 1526k| 828M 19.4M 2915M  130M|  0 2.0   0| 111   157 |4588    14k| 4928  12285 | 11   3   0   0   2|   0  1859   995  2278  920k 2151k| 826M 19.4M 2917M  129M|  0 2.0   0|52.0   237 |3091  7540 | 4928  12284 | 11   3   0   0   2|   0  4448  2330  2144 2124k 1003k| 826M 19.4M 2921M  126M|1.0 1.0   0| 135   106 |4705    14k| 4928  12284 | 11   3   0   0   2|   0   331   865     1 2344k 1024k| 826M 19.4M 2924M  122M|1.0 2.0   0| 121   118 |4074    13k| 4928  12284 | 11   3   0   0   2|   0   249   953     1 1572k 1624k| 827M 19.4M 2926M  120M|1.0 2.0   0|87.0   190 |3231    11k| 4928  12284 | 11   3   0   0   2|   0    98   530     1  916k  788k| 827M 19.4M 2928M  119M|  0 2.0   0|68.0  92.0 |3452  8709 | 4928  12284 | 11   3   0   0   2|   0   128   383     4 2452k 1665k| 826M 19.4M 2931M  116M|1.0 1.0   0| 132   197 |4779    14k| 4928  12284 | 11   3   0   0   2|   0   208   822     1 1552k 1328k| 827M 19.4M 2933M  114M|  0 2.0   0|97.0   156 |3762  9117 | 4928  12284 | 11   3   0   0   2|   0   133   473     1 1192k 2024k| 827M 19.4M 2934M  112M|  0 2.0   0|81.0   239 |4068    11k| 4928  12284 | 11   3   0   0   2|   0   135   414     2 2668k  584k| 827M 19.4M 2937M  109M|  0 2.0   0| 148  71.0 |4415    10k| 4928  12284 | 11   3   0   0   2|   0   174   870     4 1712k  960k| 827M 19.4M 2940M  106M|  0 2.0   0| 122   113 |4454    14k| 4928  12284 | 11   3   0   0   2|   0   182   616     2

For more detailed usage, see the dstat (1) manual.