Common CentOS system performance analysis and methods

Common CentOS system performance analysis and methods

▲Vmstat -- virtual memory statistics

Vmstat (VirtualMeomoryStatistics, virtual memory statistics) is a common tool used in Linux to monitor the overall situation of the operating system's virtual memory, processes, CPU, and so on.

General Usage of vmstat: vmstat interval times is used to sample data every interval seconds. If times is omitted, data is collected until the user stops it manually.
For example:

You can use ctrl + c to stop data collection by vmstat.

The first line shows the average value since the system was started, the second line shows the current situation, and the next line shows what happens every 5 seconds, the meaning of each column is in the header, as shown below:

▪Procs: The r column shows how many processes are waiting for the cpu, and the B column shows how many processes are waiting for IO ).

▪Memory: The swapd column shows how many disks are swapped out of the disk (Page Swap), the remaining columns show how many are idle (not used), and how many are being used as a buffer, and how much cache is being used as the operating system.

▪Swap: Display swap activity: Number of swap-in (from disk) and swap-out (to disk) per second ).

▪Io: displays the number of blocks read (bi) and write (bo) from the block device, usually reflecting the hard disk I/O.

▪System: displays the number of interruptions (in) and context switches (cs) per second.

▪Cpu: displays the percentage of cpu time spent on various operations, including Executing User code (non-kernel), executing system code (kernel), idle, and waiting for IO.

Memory Insufficiency: free memory decreases sharply, and the recovery of buffer and cacher does not help. swap partitions (swpd) are used in large quantities, page switching (swap) is frequent, and the number of read/write disks (io) increases, page disconnection (in) increases, context switching (cs) increases, and the number of processes waiting for IO (B) increases. A large amount of CPU time is used to wait for IO (wa)

▲Iostat -- used to report statistics of the central processor

Iostat is used to report central processor (CPU) statistics and input/output statistics for the entire system, adapter, tty device, disk, and CD-ROM, which by default displays the same cpu usage information as vmstat, run the following command to display the extended device statistics:

The first line displays the average value since the system was started, and then the incremental average value, each device line.

Common linux disk I/O metric abbreviations: rq is request, r is read, w is write, qu is queue, sz is size, a is verage, tm is time, svc is a service.

▪Rrqm/s and wrqm/s: Read and Write requests merged per second. "merged" means that the operating system combines multiple logical requests from the queue into one request to the actual disk.

▪R/s and w/s: the number of Read and Write requests sent to the device per second.

▪Rsec/s and wsec/s: Number of Read and Write sectors per second.

▪Avgrq-sz: number of requested sectors.

▪Avgqu-sz: Number of requests waiting in the device queue.

▪Await: the time spent by each IO request.

▪Svctm: the actual request (service) time.

▪% Util: the percentage of time for at least one active request.

▲Dstat-System Monitoring Tool

Dstat displays cpu usage, disk I/O, network packet sending, and page feed. The output is color and readable. It is more detailed and intuitive than the input of vmstat and iostat. You can directly enter the command when using it. Of course, you can also use specific parameters.

Dstat-cdlmnpsy

▲Iotop -- real-time monitoring tool for LINUX Processes

The iotop command is used to display hard disk I/O commands. The interface style is similar to the top command. It can display the specific process that generates the I/O load. Is a top tool used to monitor disk I/O usage. It has a similar UI to top, including PID, user, I/O, process and other related information.

You can use iotop-bod interval in non-interactive mode to view the I/O of each process. You can use pidstat and pidstat-d instat.

▲Pidstat-Monitor System Resources

Pidstat is mainly used to monitor system resources occupied by all or specified processes, such as CPU, memory, device IO, task switching, and threads.

Usage: pidstat-d interval; pidstat can also be used to count CPU usage information: pidstat-u interval; Statistical memory information: Pidstat-r interval.

▲Top

The summary area of the top command displays the system performance information in five aspects:

1. Load: time, number of login users, and average system load;

2. Processes: running, sleeping, stopping, and botnets;

3. cpu: User, core, NICE, idle, waiting for IO, interrupted, etc;

4. Memory: total, used, idle (system angle), buffer, and cache;

5. Swap partition: total, used, idle

The default display of the task area is process ID, valid user, process priority, NICE value, virtual memory used by the process, physical memory and shared memory, Process status, CPU usage, and memory usage, accumulated CPU time and process command line information.

▲Htop

Htop is an interactive process viewer in Linux. It is a text-mode application (in the console or X terminal) that requires ncurses.

Htop allows you to perform interactive operations. It supports color themes, horizontal or vertical scrolling of the Process List, and supports mouse operations.

Compared with top, htop has the following advantages:

▪You can scroll through the process list either horizontally or vertically to view all processes and complete command lines.

▪It is faster than top at startup.

▪You do not need to enter a process number when killing a process.

▪Htop supports mouse operations.

▲Mpstat
Mpstat is short for Multiprocessor Statistics and is a real-time system monitoring tool. Its report and CPU statistics are stored in the/proc/stat file. In a multi-CPUs system, it can not only view the average status information of all CPUs, but also view information about specific CPUs. Common usage: mpstat-p all interval times.

▲Netstat

Netstat is used to display statistics related to IP, TCP, UDP, and ICMP protocols. It is generally used to check the network connection of each port on the local machine.

▲Common usage:

Netstat-npl can check whether the port you want to open has been enabled.

Netstat-rn prints route table information.

Netstat-in provides interface information on the system, prints the MTU of each interface, number of input groups, input errors, number of output groups, output errors, conflicts, and the length of the current output queue.

▲Ps -- display the status of the current process

There are too many ps parameters. For details, refer to man ps. Common Methods: ps aux # hsserver; ps-ef | grep # hundsun

▪How to kill a program: ps aux | grep mysqld | grep-v grep | awk '{print $2}' xargs kill-9

▪Killing botnets: ps-eal | awk '{if ($2 = "Z") {print $4}' | xargs kill-9

▲Strace

Tracks system calls and received signals generated during program execution to help analyze exceptions encountered during program or command execution.

For example, to view which configuration file mysqld is loaded on linux, run the following command: strace-e stat64 mysqld-print-defaults>/dev/null.

▲Uptime

It can print the total running time of the system and the average load of the system. The three numbers output by the uptime command indicate the average load of the system within 1 minute, 5 minutes, and 15 minutes respectively.

▲Lsof

Lsof (list open files) is a tool used to list open files in the current system. You can use the lsof tool to view system detection and troubleshooting information in this list. Common usage:

View File System blocking lsof/boot

Check which process occupies the port number lsof-I: 3306

Check which files the user opens, lsof-u username

View the files opened by the process lsof-p 4838

View the remotely opened network link lsof-I @ 192.168.34.128

▲Perf

Perf is a built-in system performance optimization tool for Linux kernel. The advantage lies in the close integration with Linux Kernel. It can be applied to the new feature added to the Kernel first, used to view the hotspot functions and view the ratio of cashe miss, so as to help developers optimize program performance.

The basic principles of performance tuning tools such as perf and Oprofile are to sample the monitored objects. The simplest case is to sample the Objects Based on the tick interrupt, that is, trigger the sampling points within the tick interrupt, determine the context of the program at the sampling point. If 90% of the time of a program is spent on function foo (), 90% of the sampling points should fall in the context of function foo. Luck is unpredictable, but I think the above inferences are reliable as long as the sampling frequency is high enough and the sampling time is long enough. Therefore, using tick to trigger sampling, we can find out which parts of the program are most time-consuming and thus focus on analysis.

Summary: The diagram of the above commonly used performance test commands and the performance analysis tool at the beginning of the article is as follows, you can get a preliminary understanding of which performance tool (command) is used in the performance analysis process ).