Detailed output of vmstat and iostat on various UNIX Platforms

Description of iostat and vmstst on various unix Platforms

A little personal accumulation, sharing out!

Explain vmstat output in Linux:

Vmstat
Procs ----------- memory ---------- --- swap -- ----- io ---- system -- ---- cpu ----
R B swpd free buff cache si so bi bo in cs us sy id wa
0 0 100152 2436 97200 289740 0 1 34 45 99 33 0 99 0

Procs
The r column indicates the number of processes that run and wait for the cpu time slice. If the number is greater than 1 for a long time, the cpu is insufficient and the cpu needs to be increased.
Column B indicates the number of processes waiting for resources, such as waiting for I/O or memory switching.
Cpu indicates the cpu usage status
The us column shows the percentage of CPU time spent in user mode. When the value of us is high, it indicates that the user process consumes a lot of cpu time. However, if it is longer than 50%, you need to consider optimizing your program.
The sy column shows the percentage of cpu time consumed by the kernel process. Here, the reference value of us + sy is 80%. If the value of us + sy is greater than 80%, there may be insufficient CPU.
The wa column shows the percentage of CPU time occupied by IO wait. The reference value of wa is 30%. If wa exceeds 30%, the IO wait is serious, which may be caused by a large number of random access to the disk, it may also be caused by the bandwidth bottleneck of the disk or disk access controller (mainly block operations ).
The id column shows the percentage of time when the cpu is idle.
The system displays the number of interruptions during the collection interval.
The in column indicates the number of device interruptions per second observed at a certain interval.
The cs column indicates the number of context switches generated per second. If cs is much higher than the disk I/O and network information packet rate, further investigation should be conducted.
Memory
The number of memories that swpd switches to the memory swap area (k indicates ). If the value of swpd is not 0 or is relatively large, for example, if the value exceeds 100 m, as long as the value of si and so is 0 for a long time, the system performance is still normal.
Free memory in the current idle page list (k indicates)
The amount of memory that buff uses as the buffer cache. Generally, the buffer is required for reading and writing Block devices.
Cache: The amount of memory that is used as the page cache. Generally, it is used as the cache of the file system. If the cache is large, it indicates that there are many files in the cache. If the bi in IO is small at this time, it indicates that the file system is more efficient.
Swap
Si refers to the number of switches from memory to memory swap zone.
So the number of memory entries in the memory swap zone.
IO
The total amount of data that bi reads from Block devices (read disk) (kb per second ).
Total data written to bo Block devices (disk write) (kb per second)
Here we set the reference value of bi + bo to 1000. If the value of bi + bo is greater than 1000 and the wa value is large, we should consider Disk Load Balancing. We can use iostat output for analysis.

Explain vmstat output in Solairs System

Vmstat 5 5
Procs memory page disk faults cpu
R B w swap free re mf pi po fr de sr s1 s1 -- in sy cs us sy id
0 0 0 704968 207640 0 1 2 0 0 0 0 0 0 0 302 6 31 0 0 100
0 0 0 705048 206728 1 2 0 0 0 0 0 0 0 0 302 4 34 0 0 100
0 0 0 705048 206728 0 0 0 0 0 0 0 0 0 0 302 3 35 0 0 100
0 0 0 705048 206728 0 0 0 0 0 0 0 0 0 0 301 3 34 0 0 100
0 0 0 705048 206728 0 0 0 0 0 0 0 0 0 0 302 6 35 0 0 100
Procs
The r column indicates the number of processes waiting in the running queue.
Column B indicates the number of processes waiting for resources, such as waiting for I/O or memory switching.
The w column indicates the processes that can enter the running queue but are exchanged.
Cpu indicates the cpu usage status
The us column shows the percentage of CPU time spent in user mode. A unix process can be executed either in user mode or in system (kernel) mode. In user mode, a process is executed in its own application code and does not require kernel resources to compute, manage memory, or set variables.
The sy column details the percentage of time the CPU spends executing a process in the system mode. This includes the CPU resources consumed by kernel processes (kprocs) and other processes that need to access kernel resources. If a process requires kernel resources, it must execute a system call and switch to the system mode to make the resource available. For example, read or write operations on a file require kernel resources to open the file, find a specific location, and read or write data, unless the memory ing file is used. Here, the reference value of us + sy is 80%. If the value of us + sy is greater than 80%, there may be insufficient CPU.
The id column shows the percentage of CPU idle or waiting time when no pending ephemeral disk I/O is available.
Faults shows the number of interruptions during the collection Interval
The in column indicates the number of device interruptions per second observed at a certain interval. Iostat command output is more informative.
The sy column indicates the number of system calls per second observed at a certain interval. Through explicit system calls, user processes can use resources. These calls indicate that the kernel executes the operations of the calling thread and exchanges data between the kernel and the process. Because the workload and application changes a lot and different calls execute different functions, it is impossible to define the number of system calls per second. Set the reference value to 10000, which exceeds 10000.
The cs column indicates that when cs is much higher than disk I/O and network information packet rate, further investigation should be conducted.
Memory
Swap current available memory swap Zone capacity (k indicates ).
Free memory in the current idle page list (k indicates)
Page
The re column indicates the number of recycled pages.
The mf column indicates a small error count.
The pi column indicates the number of pages (k indicates ).
The po column indicates the number of pages (k indicates ).
The fr column indicates the number of free pages (k indicates ).
The de column indicates the number of missed pages read in advance.
The sr column indicates the page count scanned by the clock algorithm.

Solairs iostat

Iostat-xtc 5 2
Extended device statistics tty cpu
Device r/s w/s kr/s kw/s wait actv svc_t % w % B tin tout us sy wt id
Sd1 0.3 0.0 3.4 0.1 0.0 0.0 13.2 0 0 0 0 0 0 100
Sd15 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
Nfs1 0.0 0.0 0.0 0.0 0.0 0.0 0 0
Extended device statistics tty cpu
Device r/s w/s kr/s kw/s wait actv svc_t % w % B tin tout us sy wt id
Sd1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 77 0 0 0 100
Sd15 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
Nfs1 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Tty and CPU usage reports
Tin shows the total number of characters read by the system for all tty.
Tout shows the total number of characters written by the system to all tty.
Us displays the percentage of CPU usage when user-level applications are executed at the default priority level.
Sy shows the percentage of CPU usage when user-level applications perform lower-priority execution.
Wt shows the CPU usage generated during system-level (kernel) execution.
Id shows the percentage of time when the system has incomplete disk I/O requests during idle CPU.
Disk usage report
The name of the hard disk device, including the nfs device.
R/s indicates the number of reads per second
W/s indicates the number of writes per second
Kr/s indicates the number of K Bytes read per second
Kw/s indicates the number of K bytes written per second
Wait indicates the average number of transmission waits to be processed, which is equivalent to the length of the read/write waiting queue.
Actv indicates the average number of transmissions removed from the read/write wait queue for processing.
Svc_t indicates the average time consumed by each Hard Disk transmission, in milliseconds. Here, the reference value is 30 ms. exceeding the value indicates that the hard disk is overloaded.
% W indicates the percentage of waiting time consumed by each transmission.
% B indicates the percentage of Hard Disk busy time. Here we set the reference value to 5%. If the value exceeds 5%, the hard disk load is too large. It is best to take the following steps:
1) Adjust the application to make it more efficient to use disk I/o. You can modify the disk queue and use the cache of the application server.
2) Distribute the file system to two or more disks and use the Strip feature of volume manager/disksuite.
3) add system parameter values, such as inode cache and ufs_ninode.

Explain vmstat in AIX:

# Vmstat 5 20
Kthr memory page faults cpu
---------------------------------------------------------------
R B avm fre re pi po fr sr cy in sy cs us sy id wa
1 1 244528 1227 0 3 56 151 0 404 1420 4 2 88 6
2 6 244538 1241 0 39 148 2424 3511 0 904 4660 49 12 2 38
5 3 244540 791 0 2 221 3937 6077 1183 6605 24619 44 34 1 21
1 7 245326 1009 0 11 331 1071 1934 0 884 3788 23 25 0 52

The analysis is as follows:
Kthr
The kthr list shows the average value of the number of kernel threads in various queues at the sampling interval per second.
The r column represents the average number of running kernel threads, including running threads and waiting CPU threads. If this number is greater than the number of CPUs, at least one thread will wait for the CPU. The more threads wait for the CPU, the more likely it will affect the performance.

Column B indicates the average number of kernel threads in the VMM wait queue per second. This includes threads waiting for file system I/O, or threads temporarily suspended due to memory loading control. Here, the reference value is 2. If the value is greater than 2, the number of threads in the blocked column is too large.
[B]! Too many blocked threads
CPU
The us column shows the percentage of CPU time spent in user mode. A unix process can be executed either in user mode or in system (kernel) mode. In user mode, a process is executed in its own application code and does not require kernel resources to compute, manage memory, or set variables.
The sy column details the percentage of time the CPU spends executing a process in the system mode. This includes the CPU resources consumed by kernel processes (kprocs) and other processes that need to access kernel resources. If a process requires kernel resources, it must execute a system call and switch to the system mode to make the resource available. For example, read or write operations on a file require kernel resources to open the file, find a specific location, and read or write data, unless the memory ing file is used. Here, the reference value of us + sy is 80%. If the value of us + sy is greater than 80%, there may be insufficient CPU.
The id column shows the percentage of CPU idle or waiting time when no pending ephemeral disk I/O is available.
The wa column shows in detail the percentage of CPU idle on ephemeral disk I/O and NFS attached disks. The reference value here is 25%. a value greater than 25% may indicate that the disk subsystem may not be properly balanced, or this may be the result of intensive disk workloads.
[Us] + [sy]! IO imbalance
[Wa] Normal
Fault
The in column indicates the number of device interruptions per second observed at a certain interval. Iostat command output is more informative.
The sy column indicates the number of system calls per second observed at a certain interval. Through explicit system calls, user processes can use resources. These calls indicate that the kernel executes the operations of the calling thread and exchanges data between the kernel and the process. Because the workload and application changes a lot and different calls execute different functions, it is impossible to define the number of system calls per second. Set the reference value to 10000, which exceeds 10000.
The cs column indicates that when cs is much higher than disk I/O and network information packet rate, further investigation should be conducted.
[Sy] Normal
Memory
The avm column indicates the number of active virtual memory pages (in 4 K) that exist when the vmstat sample is collected ). Ideally, avm should be smaller than the total number of RAM. If not, some Virtual Memory Page scheduling may occur. The number of page scheduling depends on the difference between the two values. Remember, the concept of virtual memory is to provide us with the ability to address larger than the actual memory capacity (some in RAM memory and others in paging space ). However, if the virtual memory is much larger than the real memory, excessive page scheduling may occur, resulting in latency. If avm is smaller than RAM, page scheduling is triggered when the file page is filled in RAM. In this case, adjusting the values of minperm, maxperm, and maxclient can reduce the page scheduling volume of the page space.
The fre column displays the average number of idle memory pages (in 4 K ). The minimum number of pages that VMM saves on the idle list is determined by the minfree parameter. The reference value here is the default value of 720 for maxfree. If the value is smaller than 720, the memory system is bumpy.
[Fre] Normal
[Avm] avm value is about 958 MB.
Page
The pi column describes in detail the number of pages transferred from the paging space. Paging space is part of the virtual memory residing on the disk. When the memory usage exceeds the limit, it is used as an overflow. The paging space is composed of logical volumes used to store the workgroup pages stolen from the real memory. When a process accesses a page to steal a page, a page failure occurs. This page must be read into the memory from the page space. Here, the pi reference value is 5. If the value is greater than 5, the memory is insufficient.
The po column shows the number of pages (rate) that are transferred to the page space ). No matter when a page of the work memory is stolen, if it does not reside in the paging space or has been modified, it will be written into the paging space. If it is not accessed again, it will remain on the page scheduling device until the process is terminated or the space is abandoned. If the subsequent address reference contained in the fault page causes a page failure, these pages will be individually transferred by the system. When a process is terminated normally, any page space allocated to the process will be released.
The fr column indicates the number of pages released per second based on the page replacement algorithm within a certain interval. When the VMM Page Replacement routine scans the Page Frame Table (PFT), it uses some conditions to select the Page to be stolen and insert it to the Free List of available memory frames.
The sr column indicates the number of pages inspected per second based on the page replacement algorithm within a certain interval. The page replacement algorithm may have to scan many page frames before it can steal enough pages to meet the needs of the page replacement thread.
The cy column indicates how many times the Page Replacement code scanned PFT per second. You do not need to scan PFT completely when inserting an idle list. Because all vmstat fields are reported as integers, this field is usually 0.
The reference value of fr: sr is 3. If the value exceeds 3, the memory is insufficient.
[Pi]! Insufficient memory
[Fr: sr] Normal

AIX command iostat

# Iostat hdisk1 hdisk2 5 5

Tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 0.3 0.2 0.4 99.4 0.1

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
Hdisk2 0.1 1.0 0.2 45090 51075
Hdisk1 0.0 0.0 0.0 0 0

Tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 70.6 45 54 99.4 0.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
Hdisk2 0.0 0.0 0.0 0 0
Hdisk1 0.0 0.0 0.0 0 0

Tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 70.6 0.0 0.4 99.6 0.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
Hdisk2 0.0 0.0 0.0 0 0
Hdisk1 0.0 0.0 0.0 0 0
Analysis result
Tty and CPU usage reports
Tin shows the total number of characters read by the system for all tty.
Tout shows the total number of characters written by the system to all tty.
% User shows the percentage of CPU usage generated during user-level (Application) execution.
% Sys displays the percentage of CPU usage generated during system-level (kernel) execution.
% Idle shows the percentage of time when the CPU is idle and the system does not complete disk I/O requests.
% Iowait shows the percentage of time when the system has incomplete disk I/O requests during idle CPU.
[User] [+] [sys]! Insufficient CPU capacity
[Wait] Normal
Disk usage report
% Tm_act indicates the percentage of time when the physical disk is active (bandwidth usage of the drive ).
Kbps indicates the amount of data transmitted (read or write) to the drive in KB per second.
Tps indicates the number of transfers that are output to the physical disk per second. One transmission is an I/O Request to a physical disk. Multiple logical requests can be combined as a single I/O Request to the disk. Transmission has an uncertain size.
The total number of KB read by Kb_read.
The total number of KB written by Kb_wrtn.
The Disk Load Balancing evaluation is normal.
Bandwidth usage evaluation bandwidth: 0 Kbps

Hp-UNIX Command vmstat

# Vmstat 5 5
Procs memory page faults cpu
R B w avm free re at pi po fr de sr in sy cs us sy id
2 0 0 52260 24324 9 4 0 0 0 1 109 81 2 1 97
2 0 0 52260 24285 0 0 0 0 0 0 110 207 90 0 99
1 0 0 56211 24285 0 0 0 0 0 106 192 80 0 100
1 0 0 56211 24285 0 0 0 0 0 107 152 76 0 100
2 0 0 56552 24285 0 0 0 0 0 106 154 72 0 100

Analysis result
Procs
The r column indicates the number of processes waiting in the running queue.
Column B indicates the number of processes waiting for resources, such as waiting for I/O or memory switching.
The w column indicates the processes that can enter the running queue but are exchanged.
[B] Normal
Cpu indicates the cpu usage status
The us column shows the percentage of CPU time spent in user mode. A unix process can be executed either in user mode or in system (kernel) mode. In user mode, a process is executed in its own application code and does not require kernel resources to compute, manage memory, or set variables.
The sy column details the percentage of time the CPU spends executing a process in the system mode. This includes the CPU resources consumed by kernel processes (kprocs) and other processes that need to access kernel resources. If a process requires kernel resources, it must execute a system call and switch to the system mode to make the resource available. For example, read or write operations on a file require kernel resources to open the file, find a specific location, and read or write data, unless the memory ing file is used. Here, the reference value of us + sy is 80%. If the value of us + sy is greater than 80%, there may be insufficient CPU.
The id column shows the percentage of CPU idle or waiting time when no pending ephemeral disk I/O is available.
[Us] + [sy] Normal
Fault displays the number of interruptions during the collection Interval
The in column indicates the number of device interruptions per second observed at a certain interval. Iostat command output is more informative.
The sy column indicates the number of system calls per second observed at a certain interval. Through explicit system calls, user processes can use resources. These calls indicate that the kernel executes the operations of the calling thread and exchanges data between the kernel and the process. Because the workload and application changes a lot and different calls execute different functions, it is impossible to define the number of system calls per second. Set the reference value to 10000, which exceeds 10000.
The cs column indicates that when cs is much higher than disk I/O and network information packet rate, further investigation should be conducted.
[Sy] Normal
Memory
The number of virtual memory activated by avm (k indicates ).
Free memory in the current idle page list (k indicates)
[Free] Normal
Page
The re column indicates the number of recycled pages.
The at column indicates the address translation error count.
The pi column indicates the number of pages (k indicates ).
The po column indicates the number of pages (k indicates ).
The fr column indicates the number of free pages (k indicates ).
The de column indicates the number of missed pages read in advance.
The sr column indicates the page count scanned by the clock algorithm.
[Po] Normal
[Fr: sr] Normal

Remove iostat in Hp-unix

# Iostat 5 5

Device bps sps msps

C0t6d0 0 0.0 1.0

C0t6d0 2 1.0 1.0

C0t6d0 2 0.8 1.0

C0t6 D0 4 2.0 1.0

C0t6d0 2 1.0 1.0
Analysis result
Buffer Performance Analysis
Device: the device name measured by iostat.
Bps: the number of K Bytes transmitted by the device per second.
Sps: the number of times the device is retrieved per second.
Msps: Average time spent on each retrieval, in milliseconds.
[Bps] Normal

 

 

 

 

Vmstat command
The first tool to be used is the vmstat command, which quickly provides brief information about various system resources and related performance issues.

The vmstat Command reports statistics about Kernel threads, includes kernel threads in the running and waiting queue, in memory, in page scheduling, on disk, interrupted, system calls, context switching, and CPU activity. The reported CPU activity is divided into user, system, idle time, and percentage of disk I/O waiting.
Note: If you use the vmstat command without any options, or with only time interval and (optional) Count parameters, such as vmstat 2 10, the number in the first line is the average value since the system reboot.
As a CPU Monitor, the vmstat command is better than the iostat command, because the vmstat command is scroll, making it easier to scan the output of each report line, and if many disks are connected to the system, this reduces the overhead. The following example helps you identify situations where a program is out of control or when the CPU is too intensive to run in a multi-user environment.
# Vmstat 2
Kthr memory page faults cpu
---------------------------------------------------------------
R B avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 22478 1677 0 0 0 0 0 188 1380 57 32 0 10
1 0 22506 1609 0 0 0 0 0 214 1476 48 37 0 16
0 0 22498 1582 0 0 0 0 0 248 1470 55 36 0 9

2 0 22534 1465 0 0 0 0 0 238 903 239 77 23 0 0
2 0 22534 1445 0 0 0 0 0 209 1142 72 28 0 0
2 0 22534 1426 0 0 0 0 0 189 1220 74 26 0 0
3 0 22534 1410 0 0 0 0 0 255 1704 70 30 0 0
2 1 22557 1365 0 0 0 0 0 383 977 72 28 0 0

2 0 22541 1356 0 0 0 0 0 237 1418 63 33 0 4
1 0 22524 1350 0 0 0 0 0 241 1348 52 32 0 16
1 0 22546 1293 0 0 0 0 0 217 1473 51 35 0 14

This output shows the effect of introducing a program into a busy multi-user system in an endless loop. The first three reports (Deleted summaries) indicate that the system is balanced among 50-55% users, 30-35% systems, and 10-15% I/O waits. When the cyclic program starts running, all available CPU cycles are consumed. Because the loop program does not perform I/O, it can occupy all the cycles that have not been used due to I/O wait. Worse, this means that when a useful process abandons the CPU, there is always a process ready to take over the CPU. Because the priority of the loop program is the same as that of all other foreground processes, when another process becomes allocable, it does not have to discard the CPU. The program runs for about 10 seconds (five reports), and then the activities reported by the vmstat command are restored to normal mode.

The best use is to allow the CPU to work in 100% of the time. This is suitable for single-user systems without the need to share the CPU. In general, if the us + sy time is lower than 90%, it is not considered that the CPU is restricted for a single user system. However, if the us + sy time in a multi-user system exceeds 80%, the process may spend time waiting in the running queue. Response time and throughput will be compromised.

To check whether the CPU is a bottleneck, consider the four cpu columns and two kthr (kernel thread) columns in the vmstat report. It is also worthwhile to view the fault column:

Cpu
The percentage of CPU time used during this interval. The cpu column is as follows:

Us
The us column shows the percentage of CPU time spent in user mode. UNIX processes can be executed either in user mode or in system (kernel) mode. In user mode, a process is executed in its own application code and does not require kernel resources to compute, manage memory, or set variables.

Sy
The sy column details the percentage of time the CPU spends executing a process in the system mode. This includes the CPU resources consumed by kernel processes (kproc) and other processes that need to access kernel resources. If a process requires kernel resources, it must execute a system call and switch to the system mode to make the resource available. For example, read or write operations on a file require kernel resources to open the file, find a specific location, and read or write data, unless the memory ing file is used.

Id
The id column shows the percentage of idle CPU or waiting time when no ephemeral disk I/O is available. If no thread is available for execution (the running queue is empty), the system will assign a thread called wait (also known as idle kproc ). In an SMP system, each processor has a wait thread that can be assigned. Reports generated by ps commands (with the-k or-g 0 option) are identified as kproc or wait. If the ps report shows that the total time of this thread is relatively high, this indicates that there is an important time period, and no other threads are prepared to run on the CPU or wait for execution. Therefore, the system spends most of its time idle and waiting for new tasks.

Wa
The wa column details the percentage of idle CPU time when the ephemeral disk I/O is temporarily mounted and the NFS disk is loaded. If at least one unfinished disk I/O is run in wait, the time is classified as I/O wait time. Unless the process uses abnormal I/O, I/O requests to the disk will cause the called process to be blocked (or sleep) until the request is complete. Once the process's I/O request is complete, the process is put into the running queue. If I/O is completed quickly, the process can use more CPU time.

A value of over 25% wa may indicate that the disk subsystem may not be properly balanced, or this may also be the result of intensive disk workloads.