Go Linux Common Performance detection commands

First, uptime
The results of the uptime command include how long the server has been running, how many logged-in users, and the overall evaluation of server performance (load average). The load average values are recorded for the last 1 minutes, 5 minutes, and 15 minute intervals, and load average is not a percentage, but the number of processes waiting to be executed in the queue. If the process requires CPU time to be blocked (meaning that the CPU does not have time to handle it), the load average value will increase. On the other hand, if each process can get access to the CPU immediately, this value will be reduced.
The best value for the load average under up kernel is 1, which means that each process can be processed by the CPU at once, and, of course, the lower will not be a problem, only a waste of a portion of the resources. But this value is also different between different systems, such as a single CPU workstation, load average 1 or 2 is acceptable, and in a multi-CPU system This value should be divided by the number of physical CPUs , assuming that the number of CPUs is 4, and load Average is 8 or 10, and the result is at 2多 points.

You can use uptime to determine whether a performance problem occurs on the server or on the network. For example, if a network application is running at an unsatisfactory performance, run uptime to check if the system load is higher, if not the problem is more likely to occur on your network.
Second, top
The top command shows the actual CPU usage, which by default shows the CPU-intensive task information on the server and refreshes every 5 seconds. You can categorize them in a number of ways, including PID, time, and memory usage.

The following is an introduction to the output values:

Reference PID: Process identity
User name of the process owner
PRI: Priority of the process
Ni:nice level
Size: The amount of memory the process consumes (code + data + stack)
RSS; The amount of physical memory used by the process
SHARE; The number of shared memory for this process and other processes
STAT: Status of the process: s= hibernation, r= running state, t= stop state, d= interrupt hibernation, z= zombie status
%CPU: Shared CPU usage
%MEM; shared physical memory
Time: How long the process consumes CPU
Command: To start a task (including parameters)

Process priority and nice level
The process priority is a parameter that determines the process is prioritized by the CPU, and the kernel adjusts the value as needed. The nice value is a restriction on priority. The value of the process priority cannot be less than the nice value. (The lower the Nice value the higher the priority)
Process priority cannot be changed manually, but only by changing the nice value to indirectly adjust the process priority level. If a process is running too slowly, you can allocate more CPU resources to it by specifying a lower nice value. Of course, this means that some other processes will be allocated less CPU resources and run slower. Linux supports a nice value with a range of 19 (low priority) to 20 (high priority), and the default value is 0. If you need to change the nice value of a process to a negative number (high priority), you must log on to the root user using the SU command. Here are some examples of commands to adjust nice values.
Start program xyz with nice value-5

#nice –n-5 XYZ

Change the nice value of a program that is already running

#renice level PID

Change the nice value of the process with PID 2500 to 10

#renice 10 2500

Zombie Process
When a process is finished, it usually takes some time to complete all tasks (such as closing open files) before it ends, and in a very short time, the status of this process is zombie state. After the process has completed all the shutdown tasks, it submits the information it closed to the parent process. In some cases, a zombie process cannot shut itself down, when the process state is Z (zombie). You cannot use the KILL command to kill a zombie process because it is already marked as "dead". If you can't get rid of a zombie process, you can kill its parent process, and the zombie process disappears. However, if the parent process is the INIT process, you cannot kill the init process because INIT is an important system process, in which case you can only get rid of the zombie process by restarting the server at once. You also have to analyze why applications can cause zombies.
Third, Iostat
Iostat is part of the Sysstat package. Iostat shows the average CPU time since the system was booted (similar to uptime), which also shows the use of the disk subsystem, IOSTAT can be used to monitor CPU utilization and disk utilization.

CPU utilization is divided into four parts:

CPU usage for reference%user:user level (application)
%nice: CPU Usage for the user level with nice priority
%sys:system level (kernel) CPU usage
%idle: Idle CPU resource condition

The disk occupancy rate has the following sections:

Reference device: Block device name
Tps: The number of devices per second that the device transmits (I/O requests per second). Multiple individual I/O requests can be composed of one transport operation, because a transfer operation can be of different capacity.
BLK_READ/S, BLK_WRTN/S: The number of blocks that the device reads and writes per second. The block may be of different capacity.
Blk_read, Blk_wrtn: The total number of block devices that have been read and written since the system started.

The size of the block
The block may be of different capacity. The size of the block is generally 1024, 2048, 4048byte. Available through TUNE2FS or DUMPE2FS:

references [[email protected] ~]# tune2fs-l/dev/hda1|grep ' Block size '
Block size:4096
[[email protected] ~]# dumpe2fs-h/dev/hda1|grep ' Block size '
DUMPE2FS 1.35 (28-feb-2004)
Block size:4096

Iv. Vmstat
The Vmstat command provides monitoring of information such as processes, memory, page I/O blocks, and CPUs, Vmstat can display the average or sampled values of the test results, and the sampling mode can provide a monitoring result of different frequencies over a sampling period.

Note: In the sampling mode, you need to consider the possible errors in data collection, and set the sampling frequency to a lower value to minimize the effect of the error.
Here's a look at what each column means

Reference process (procs)
R: Number of processes waiting for run time
B: Process in non-disruptive sleep state
W: Processes that are swapped out but can still be run, this value is calculated
MEMORYSWPD: Number of virtual memory
Free: The number of idle memory
Buff: The amount of memory used as a buffer
Swap
Si: Quantity exchanged from hard disk
So: The number of swapped to the hard drive
· Io
BI: Number of blocks output to a single block device
Bo: Number of blocks accepted from a single block device
system
In: Number of interrupts per second, including clock
CS: Number of context switches that occur per second
CPU (percent of the total CPU run time)
US: Time for non-kernel code to run (user time, including nice time)
SY: Kernel code run time (System time)
ID: Idle time, in the kernel version prior to Linux 2.5.41, this value includes I/O wait time;
WA: Time to wait for I/O operation, this value is 0 in the kernel version prior to Linux 2.5.41

The Vmstat command provides a number of additional parameters, and some of the most useful parameters are listed below:

Reference • m: Show memory utilization of the kernel
A: Display memory page information, including active and inactive memory pages
• N: Displays the header line, which is useful when using sampling mode and outputting command results to a file. For example, Root#vmstat–n 2 10 displays 10 output results at 2-second frequency
• Vmstat provides statistics on I/O results when using the-p {partition}

V, PS and Pstree
The PS and Pstree commands are the most common basic commands for system analysis, and the PS command provides a list of the processes that are running, depending on the parameters attached to the command. For example, the ps–a command lists all processes and their corresponding process IDs (PID), and the PID of the process must be understood before using some other tool, such as Pmap or Renice.
On a system running Java applications, the output of the Ps–a command can easily exceed the display of the screen, which makes it difficult to get complete information for all processes. At this point, you can use the Pstree command to display all process information in a tree structure and to integrate the child process information. The Pstree command is useful for analyzing the source of a process.

Liu, Numastat
As the NUMA architecture continues to evolve, such as eserver xseries 445 and its successor EServer xseries 460, NUMA architectures are now becoming the mainstream of enterprise data centers. However, the NUMA architecture faces new challenges in performance tuning, such as the problem of memory allocation that is not interesting before NUMA systems, and the Numastat command provides a tool for monitoring NUMA architectures. The Numastat command provides a comparison of local memory with remote memory usage and memory usage for each node. The Numa_miss column displays local memory for which the allocation failed, and the Numa_foreign column displays information that allocates remote memory (slow access), and excessive calls to remote memory will increase the system's latency and affect the performance of the system as a whole. Enabling processes running on one node to access local memory will greatly improve the performance of the system.

※ The system I use does not support NUMA architecture, which is the original document.
VII. SAR
The SAR program is also part of the Sysstat installation package. The SAR command is used to collect, report, and save information about the system. The SAR command consists of three applications: SAR, using and displaying data, SA1 and SA2, for collecting and storing data. By default, automatic collection and analysis is added to the crontab:

references [[email protected] ~]# Cat/etc/cron.d/sysstat
# Run System Activity Accounting tool every minutes
*/10 * * * * ROOT/USR/LIB/SA/SA1 1 1
# Generate a daily summary of process accounting at 23:53
* * * root/usr/lib/sa/sa2-a

The data generated by the SAR command is saved in the/var/log/sa/directory, the data is saved according to the time, and the corresponding performance data can be queried according to the time.
You can also use the SAR to get a real-time execution result from the command line, which can include CPU utilization, memory pages, network I/O, and so on. The following command indicates a 5-time interval of 3 seconds in SAR execution:

Viii. Free
The free command displays all memory usage for the system, including idle memory, used memory, and swap memory space. The free command display also includes information about the caches and buffers used by the kernel.
when using the free command, it is necessary to remember the memory structure of Linux and the management methods of virtual memory, such as the limit of the amount of idle memory, and the use of swap space does not mark the occurrence of a memory bottleneck.

Parameters useful for the free command:

The references-b,-k,-m and-G respectively follow bytes, kilobytes, megabytes, gigabytes display results.
-l difference shows low and high memory
-c {count} shows the number of free outputs

Nine, Pmap
The PMAP command shows the amount of memory used by one or more processes, and you can use this tool to determine which process on the server is consuming too much memory to cause a memory bottleneck.

Ten, Strace
Strace intercepts and records the system invocation information of the process, and also includes the command signals that the process accepts. This is a useful diagnostic and debugging tool that can be used by system administrators to solve program problems by Strace.
command format, you need to specify the process ID that needs to be monitored. This is used more for developers.

Strace-p <pid>

Xi. Ulimit
You can control the use of system resources through Ulimit. See previous logs: Use Ulimit and proc to adjust system parameters
12, Mpstat
The Mpstat command is also part of the Sysstat package. The Mpstat command is used to monitor the situation of each available CPU in a multi-CPU system. The Mpstat command shows the operation of each CPU or all CPUs, as well as the use of parameters to monitor sampling results of a certain frequency, as in the case of the Vmstat command.

13. Appendix
This document intercepts and modifies the Redbook from IBM tuning Red Hat Enterprise Linux on IBM eserver xseries Servers.

Go Linux Common Performance detection commands