AIX system resource Detection

Performance Optimization of computer systems is a comprehensive topic that involves analysis of system resources and application environments (such as applications and network environments. This article will give a brief introduction to how to detect the performance of the RS/6000 system and find out the resource bottlenecks that cause performance degradation.
The so-called system resources include memory,CPUAnd input/output devices. You can use the following command to clearly understand the load of RS/6000 system resources:

Vmstat
Svmon
PS
Tprof
Iostat
Netpmon
Filemon
Note:
Use commandsVmstatInstall software packages before iostat and SAR.

Bos. Acct
Install the software package using command PS

Bos. rte. Control
Install the software package before running the tprof, svmon, netpmon, and Filemon commands.

Perfagent. Tools
We canAIXFind and install the software package on the first disk of the system disk. This article also references the commands vmtune and schedtune. You need to install the software package before using it.

Bos. ADT. Samples

Memory bottleneck
This section describes the following commands used to check the memory usage of the system. Through these commands, you can clearly understand the usage and allocation of the system memory and check whether memory resources have bottlenecks.
1.Vmstat
VmstatCommand is used to count and display the Core Thread, virtual memory, hard disk, interrupt andCPU.
Run commands
#Vmstat2 10
Note: When PI and Po are consecutive non-zero values, the system running speed will decrease.
Pi the number of pages (4 kb/Page) transferred from the page space (paging space) to the memory per second );
The number of pages that po calls from memory to paging space per second;
When the number of memory pages requested by a process exceeds the number of available pages in the memory, some memory pages are swapped out and stored in paging space, you can call it into the memory later. Because the page access speed in the page space is much lower than the access speed to the Memory Page, frequent page feed will reduce the system operation speed.

Note: When the FR and Sr values increase, the memory load of sample increases.
Fr indicates the memory idle sample or the memory page to be released by a process;
Sr indicates the number of idle pages to be checked for the FR count;
Fr/Sr ratio: 1: 4 sample indicates that each time a memory page is released, four pages need to be checked.
When Po * sys>; FR (sys is a system parameter, the command schedtune can be used.View), The system itself is considered to be at the edge of the crash. The default value of sys is 0 on systems with MB or more memory. The sample shows that memory loading control is disabled. Otherwise, the default value is 6. The system crash sample indicates that the system uses a lot of time for memory swap-in/swap-out, but does not have enough time to process applications or work properly. At this time, some processes will be temporarily suspended, and the system will obviously slow down.
2. svmon
The svmon command counts and displays the usage of the virtual memory.
Run commands as root
# Svmon-Pau 10 | more
The output is as follows:
PID command inuse pin pgspace
13794 dtwm 1603 1 449
PID: 13794
Command: dtwm
This command displays the top 10 processes with the most memory usage, and provides a detailed sample column for each process (the preceding sample shows the description of one process ). In the sample column of each process, find the corresponding row whose type is "work" and whose description is "private,ViewThe number of pages used by pgspace (the size of each page is 4096 bytes). The number of working pages occupied by this segment (segment) in the virtual memory space, that is, the number of pages in the occupied page space. If the pgspace value continues to grow, the system may have memory leakage (Memory Leak ). Memory leakage may occur when the application cannot normally release the memory.
The preceding Command Format and output result applyAIXVersions earlier than 4.3.3. InAIX4.3.3 running commands on the System
Svmon-Pu 10 | more
The output format is basically the same as that of the previous command, but the field "pgspace" is replaced by "pgsp.
3. Ps
The ps command displays the status of the currently running process.
Run the following command:
# Ps GV | head-N 1; ps gv | egrep-V "RSS" | sort + 6B-7-n-R
In the size page, the type assigned to the process is "work", and the description is "private". The size of the memory segment (in K bytes ). The value can also be displayed through the command svmon.
The size of the memory segment (physical memory) currently occupied by the RSS process (in K bytes ), including the memory segments with type "work" and description as "private", and memory segments with type "PERS" and description as "code. This value can be displayed through the command svmon.
The size of the body segment of the trs process resident memory, including the memory segment with type "PERS" and description "code. The value can be displayed through the command svmon.
% Mem RSS % of the system physical memory (% ).
As mentioned above, the ps command can also be usedViewMemory Allocation of each process.
CPUBottleneck

The following describes how to use the commandVmstatTprof and PS check whether the system existsCPULet's briefly introduce the bottleneck.
1.Vmstat
Use commands
#Vmstat1 10
Note: The system running speed decreases when the running queue has a process waiting.
IDCPUPercentage of idle time or no I/O wait time;
WaCPUPercentage of I/O wait time;
R: Number of threads in the running queue;
If the values of ID and Wa continue to be close to 0, sampleCPUIt is in the busy status.
Next let's take a look at the field R (number of threads in the running Queue ). The more threads wait in the running queue, the greater the impact on system performance.
2. tprof
The tprof command is used to countCPUUsage.
Run the following command as the root user to identifyCPUTime:
# Tprof-x sleep 30
Run this command for 30 seconds and create a file named _ prof. All in the current directory. Within 30 seconds,CPUThe number of scheduled tasks is about 3000. The field total in the _ prof. All file is scheduled toCPUTimes. If the value of the total segment corresponding to the process is 1500, the sample shows that the process is running for 3000 times.CPU1500 times in scheduling, or half ofCPUTime. The tprof output accurately shows which process is in useCPUTime.
Example of _ prof. All file:
3. netpmon
The netpmon command is used to monitor network-related I/0 andCPU.
Run the following command as root to findCPUTime and network-related codeCPUTime:
# Netpmon-O/tmp/netpmon. Out-oCPU-V; sleep 30; trcstop
This command runs for 30 seconds and generates the file netpmon. Out in the/tmp directory. FieldCPUTime is used by the ProcessCPU
Total time,CPU% Corresponds to its percentage, NetworkCPU% Indicates the network-related code in the processCPUPercentage. As follows:
Input/output (I/O) bottleneck

1. iostat
Iostat command for StatisticsCPUUsage and tty device, hard disk and CD-ROM I/0 volume.
Run the following command:
Iostat 5 10
The 10 statistical results are displayed. The following figure shows the data at one time:
Where
% Iowait waits for local I/OCPUPercentage of idle time
% Idle is not waiting for local I/OCPUPercentage of idle time
If noCPUBut at least one process is waiting for I/O,CPUThe time attribute is marked as iowait. If the percentage of iowait time is high, the sample shows that the input/output (I/O) of the disk is the main cause of slow system operation.
% Percentage of tm_act hard disk busy
Note: The tm_act value is very high, and the sample hard disk has an I/O bottleneck.
When % tm_act (Hard Disk busy time) is very high, you may feel that the system is slowing down. In some systems, when the % tm_act value of a hard disk is 60% or higher, the system performance will be affected.
Two suggestions:
1. Observe busy and idle hard disks and move data from a busy disk to a relatively idle disk. This will reduce the I/O bottleneck caused by this.

2. Check the Memory Page feed frequency according to the method described in "memory bottleneck" above. A large number of page feed operations will increase the I/O load.
2. Filemon
The Filemon command is usedViewWhich files/logical volumes/hard disks are busy. Run the following command when the system I/O is busy:
# Filemon-u-O all-o/tmp/fmon. Out; sleep 30; trcstop
After 30 seconds, the/tmp/fmon. Out file will be generated to record the tracking information. You can find out from this file:
1. The busiest virtual memory segments (segments), logical volumes, and physical volumes;
2. Check whether the hard disk I/O is caused by application operations or frequent page change operations on the page space (paging space;
3. The most active files or logical volumes. If they are stored on busy physical volumes, you can consider moving data to a relatively idle hard disk, which helps improve system performance. The sample of the busiest section lists the file systems and I nodes corresponding to the busiest files. The mount point of the file system and inode of the file can be used with the command ncheck to find the corresponding file. This sample can be used to determine whether the I/O operation is for the file system, JFS log, or system page space.
By checking the values of the "reads" and "read sequences" fields, you can determine whether the operation reads data sequentially or randomly. When "read sequences" is close to "reads", most accesses to this file are read randomly. (Appendix A is the output example of the command Filemon)
Conclusion
Here is an extremely useful tool worth mentioning. It is the RS/6000 performance diagnostic toolbox, a software package with excellent functions for detecting system performance. Xmperf, a graphic diagnostic tool based on X-Windows, is used to detect the usage of system resources in real time. (Note: This package is notAIXThe operating system disk is provided .)
In the above text, we discussed the resource bottleneck of the RS/6000 system. The specific usage of the system commands involved can be found.AIXOnline command manual. In addition, the performance of a system is closely related to network factors and applications. There are still many methods and tools for analyzing these factors. I will not describe them here. We hope that you will gradually explore and become an expert in RS/6000 system performance analysis.
Appendix A (Filemon output file example)
---------------------------------------
Thu Aug 19 11:30:49 1999
System:AIXTexmex node: 4 machine: 000691854c00
0.369 secs in measured Interval
CPUUtilization: 9.0%
Most active files
-----------------------------------------------------------------------------
# MBS # opns # RDS # WRS file volume: inode
-----------------------------------------------------------------------------
0.1 1 14 0 Smit. log/dev/hd4: 858
0.0 1 0 13 null
0.0 2 4 0 KSh. CAT/dev/hd2: 16872
0.0 1 2 0 trace. CAT/dev/hd2: 16739
Most Active Segments
------------------------------------------------------------------------------
# MBS # RPGs # WPGs segid segtype volume: inode
------------------------------------------------------------------------------
0.1 13 0 5e93 ???
0.0 2 0 22ed ???
0.0 1 0 5c77 persistent
Most active logical volumes
------------------------------------------------------------------------------
Util # rblk # wblk kb/s volume description
------------------------------------------------------------------------------
0.06 112 0 151.9/dev/hd4/
0.04 16 0 21.7/dev/hd2/usr
Most active physical volumes
------------------------------------------------------------------------------
Util # rblk # wblk kb/s volume description
------------------------------------------------------------------------------
0.10 128 0 173.6/dev/hdisk0 N/
------------------------------------------------------------------------------
Detailed file stats
------------------------------------------------------------------------------
File:/Smit. log volume:/dev/hd4 (/) inode: 858
Opens: 1
Total Bytes xfrd: 57344
Reads: 14 (0 errs)
Read sizes (bytes): AVG 4096.0 min 4096 Max 4096 SDEV 0.0
Read times (MSEC): AVG 1.709 min 0.002 Max 19.996 SDEV 5.092
File:/dev/null
Opens: 1
Total Bytes xfrd: 50600
Writes: 13 (0 errs)
Write sizes (bytes): AVG 3892.3 min 1448 Max 4096 SDEV 705.6
Write times (MSEC): AVG 0.007 min 0.003 Max 0.022 SDEV 0.006
File:/usr/lib/NLS/MSG/en_us/KSh. Cat volume:/dev/hd2 (/usr) inode: 16872
Opens: 2
Total Bytes xfrd: 16384
Reads: 4 (0 errs)
Read sizes (bytes): AVG 4096.0 min 4096 Max 4096 SDEV 0.0
Read times (MSEC): AVG 0.042 min 0.015 Max 0.070 SDEV 0.025
Lseeks: 10
File:/usr/lib/NLS/MSG/en_us/cmdtrace. Cat volume:/dev/hd2 (/usr) inode:
16739
Opens: 1
Total Bytes xfrd: 8192
Reads: 2 (0 errs)
Read sizes (bytes): AVG 4096.0 min 4096 Max 4096 SDEV 0.0
Read times (MSEC): AVG 0.062 min 0.049 Max 0.075 SDEV 0.013
Lseeks: 8
------------------------------------------------------------------------------
Detailed VM segment stats (4096 byte pages)
------------------------------------------------------------------------------
Segment: 5e93 segtype :???
Segment flags:
Reads: 13 (0 errs)
Read times (MSEC): AVG 1.979 min 0.957 Max 5.970 SDEV 1.310
Read sequences: 1
Read seq. lengths: AVG 13.0 min 13 Max 13 SDEV 0.0
Segment: 22ed segtype :???
Segment flags: inode
Reads: 2 (0 errs)
Read times (MSEC): AVG 8.102 min 7.786 Max 8.418 SDEV 0.316
Read sequences: 2
Read seq. lengths: AVG 1.0 min 1 max 1 SDEV 0.0
Segment: 5c77 segtype: Persistent
Segment flags: pers defer
Reads: 1 (0 errs)
Read times (MSEC): AVG 13.810 min 13.810 Max 13.810 SDEV 0.000
Read sequences: 1
Read seq. lengths: AVG 1.0 min 1 max 1 SDEV 0.0

------------------------------------------------------------------------------
Detailed logical volume stats (512 byte blocks)
------------------------------------------------------------------------------
Volume:/dev/hd4 Description :/
Reads: 5 (0 errs)
Read sizes (blks): AVG 22.4 min 8 Max 40 SDEV 12.8
Read times (MSEC): AVG 4.847 min 0.938 Max 13.792 SDEV 4.819
Read sequences: 3
Read seq. lengths: AVG 37.3 min 8 Max 64 SDEV 22.9
Seeks: 3 (60.0%)
Seek dist (blks): init 6344,
AVG 40.0 min 8 Max 72 SDEV 32.0
Time to next req (MSEC): AVG 70.473 min 0.224 Max 331.020 SDEV 130.364
Throughput: 151.9 KB/sec
Utilization: 0.06

Volume:/dev/hd2 Description:/usr
Reads: 2 (0 errs)
Read sizes (blks): AVG 8.0 min 8 Max 8 SDEV 0.0
Read times (MSEC): AVG 8.078 min 7.769 Max 8.387 SDEV 0.309
Read sequences: 2
Read seq. lengths: AVG 8.0 min 8 Max 8 SDEV 0.0
Seeks: 2 (100.0%)
Seek dist (blks): init 608672,
AVG 16.0 min 16 Max 16 SDEV 0.0
Time to next req (MSEC): AVG 162.160 min 8.497 Max 315.823 SDEV 153.663
Throughput: 21.7 KB/sec
Utilization: 0.04

------------------------------------------------------------------------------
Detailed physical volume stats (512 byte blocks)
------------------------------------------------------------------------------
Volume:/dev/hdisk0 Description: N/
Reads: 7 (0 errs)
Read sizes (blks): AVG 18.3 min 8 Max 40 SDEV 12.6
Read times (MSEC): AVG 5.723 min 0.905 Max 20.448 SDEV 6.567
Read sequences: 5
Read seq. lengths: AVG 25.6 min 8 Max 64 SDEV 22.9
Seeks: 5 (71.4%)
Seek dist (blks): init 4233888,
AVG 171086.0 min 8 Max 684248 SDEV 296274.2
Seek dist (% tot blks ):
The init 48.03665,
AVG 1.94110 min 0.00009 Max 7.76331 SDEV 3.36145
Time to next req (MSEC ):
AVG 50.340 min 0.226 Max 315.865 SDEV 108.483
Throughput: 173.6 KB/sec
Utilization: 0.10