Ultra-complete finishing! Linux Performance analysis Tools Summary Collection

This article was compiled out of interest in the Linux operating system and a strong desire for underlying knowledge. This article can also be used as an indicator of basic knowledge, and the article covers all aspects of a system. If there is no perfect computer system knowledge, network knowledge and operating system knowledge, documentation tools, is impossible to fully grasp, in addition to the system performance analysis and optimization is a long-term series.

This document is mainly a combination of Linux Daniel, Netflix advanced performance architect Brendan Gregg update the Linux Performance Tuning Tool blog post, to collect Linux system performance optimization related articles collated a comprehensive article, The main is to combine the blog post on the principles involved and performance testing tools to expand the description.

Background knowledge: Having background knowledge is needed to analyze performance issues. such as the hardware cache, and then the operating system kernel. The behavior details of the application are often involved with these things, and these underlying things affect the performance of the application in unexpected ways, such as the inability of some programs to take full advantage of the cache, resulting in degraded performance. such as unnecessary calls to excessive system calls, resulting in frequent kernel/user switching and so on. Here is just for the following content of this article to do some foreshadowing, about tuning there are many things, I do not know more than know, I hope we can learn together progress.

The Performance analysis tool

First look at a picture:

Brendan Gregg is a one-time analysis of the share, all of the tools can be obtained through the man to its help document, the following is a brief introduction to the general usage:

▲vmstat--Virtual Memory Statistics

Vmstat (virtualmeomorystatistics, virtual memory statistics) is a common tool for monitoring memory in Linux, which monitors the overall situation of the operating system's virtual memory, processes, CPUs, and so on.

General usage of Vmstat: The Vmstat interval times is sampled once every interval seconds, and if the time is omitted, the data is collected until the user manually stops.
A simple example:

You can use CTRL + C to stop vmstat collecting data.

The first line shows the average of the system since startup, the second line begins to show what is happening now, and the next row shows what happens every 5 seconds, and the meaning of each column is in the head as follows:

? Procs:r This column shows how many processes are waiting for the cpu,b column to show how many processes are being interrupted by hibernation (waiting for IO).

? The MEMORY:SWAPD column shows how many blocks have been swapped out of disk (paging), and the remaining columns show how many blocks are idle (unused), how many blocks are being used as buffers, and how much of the cache is being used as the operating system.

? Swap: Shows the swap activity: how many blocks are being swapped in (from disk) and swapped out (to disk) per second.

? IO: Shows how many blocks are read (BI) and write-out (Bo) from the block device, usually reflecting the hard disk I/O.

? System: Displays the number of interrupts (in) and context switches (CS) per second.

? CPU: Displays the percentage of all CPU time spent on various operations, including executing user code (non-kernel), executing system code (kernel), idle, and waiting for IO.

Out-of-memory performance: a sharp reduction in the free memories, the recovery of buffer and Cacher also useless, a large number of swap partitions (SWPD), paging (swap) Frequent, the number of read and write disks (IO) increased, page break (in) increased, the number of context switches (CS) increased, Number of processes waiting for IO (b) increased, CPU time waiting for IO (WA)

▲iostat--for reporting CPU statistics

The Iostat is used to report input/output statistics for the central Processing Unit (CPU) statistics and the entire system, adapter, TTY device, disk, and CD-ROM, by default displaying the same CPU usage information as Vmstat, and using the following command to display extended device statistics:

The first line shows the average since the start of the system, and then shows the average of the increments, one row per device.

Common Linux Disk IO indicator abbreviation habits: RQ is Request,r is Read,w is Write,qu is Queue,sz is Size,a is Verage,tm is time,svc is the service.

? rrqm/s and wrqm/s: merged read and write requests per second, "merged" means that the operating system takes multiple logical requests from the queue and merges them into a single request to the actual disk.

? r/s and W/S: The number of read and write requests sent to the device per second.

? rsec/s and WSEC/S: Number of sectors Read and written per second.

? Avgrq–sz: The number of sectors requested.

? Avgqu–sz: The number of requests waiting in the device queue.

? Await: The time spent on each IO request.

SVCTM: Actual request (service) time.

%util: The percentage of time that an active request occupies at least one.

▲dstat--System Monitoring Tool

Dstat shows CPU usage, disk IO conditions, network packet conditions and page breaks, the output is color, readable, and more detailed and intuitive than the input of Vmstat and Iostat. When you use it, you can enter the command directly and, of course, use specific parameters.

as follows: Dstat–cdlmnpsy

▲iotop--linux process Real-time monitoring tool

The iotop command is a command that displays hard disk IO specifically, and the interface style is similar to the top command, which shows which process the IO load is specifically generated from. is a top class tool for monitoring disk I/O usage, with a UI similar to top, which includes information about PID, user, I/O, process, and so on.

Can be used in a non-interactive manner: Iotop–bod interval, viewing the I/O for each process, can use Pidstat,pidstat–d instat.

▲pidstat--Monitoring System Resource situation

Pidstat is primarily used to monitor all or specified processes that consume system resources, such as CPU, memory, device IO, task switching, threading, and so on.

How to use: Pidstat–d Interval;pidstat can also be used to count CPU usage information: pidstat–u interval; statistic memory information: pidstat–r interval.

▲top

The summary area of the top command shows five aspects of system performance information:

1. Load: Time, number of login users, system average load;

2. Process: Run, sleep, stop, zombie;

3.CPU: User state, nuclear mentality, nice, idle, wait io, interrupt, etc.;

4. Memory: Total, used, idle (System angle), buffering, caching;

5. Swap partition: Total, used, idle

The task area is displayed by default: Process ID, active user, process priority, nice value, virtual memory used by the process, physical memory and shared memory, process state, CPU utilization, memory usage, cumulative CPU time, process command line information.

▲htop

Htop is an interactive process viewer in a Linux system, and a text-mode application (in the console or X terminal) requires ncurses.

Htop allows users to interact interactively, support color themes, scroll through the list of processes horizontally or vertically, and support mouse actions.

Compared to top, Htop has the following advantages:

? You can scroll through the list of processes horizontally or vertically to see all the processes and the complete command line.

? On startup, it's faster than top.

? Process number is not required to kill the process.

? Htop supports mouse operation.

▲mpstat
Mpstat is the abbreviation of multiprocessor statistics and is a real-time system monitoring tool. Some statistical information about its report and CPU, which is stored in the/proc/stat file. In a multi-CPUs system, it not only can view the average status information of all CPUs, but also can view the information of specific CPU. Common usage: mpstat–p all interval times.

▲netstat

The netstat is used to display statistics related to IP, TCP, UDP, and ICMP protocols, and is typically used to verify the network connectivity of each port on the machine.

▲ Common usage:

NETSTAT–NPL can see if the port you want to open is already open.

NETSTAT–RN Print routing table information.

The Netstat–in provides interface information on the system, prints the MTU of each interface, the number of input packets, input errors, output groupings, output errors, conflicts, and the length of the current output queue.

▲ps--shows the status of the current process

PS parameter too many, the specific use method can refer to man PS, common method: PS aux #hsserver; ps–ef |grep #hundsun

? How to Kill a program: PS aux | grep mysqld | Grep–v grep | awk ' {print $} ' Xargs kill-9

? Kill Zombie Process: Ps–eal | awk ' {if ($ = = "Z") {print $4}} ' | Xargs kill-9

▲strace

Track system calls and received signals generated during program execution to help analyze anomalies encountered during program or command execution.

Example: To see what configuration file mysqld load on Linux, you can run the following command: Strace–e stat64 mysqld–print–defaults >/dev/null

▲uptime

To be able to print how long the system has been running and the average load of the system, the three digits of the last output of the uptime command are 1 minutes, 5 minutes, and the average load of the system within 15 minutes.

▲lsof

Lsof (list open files) is a tool that lists open files for the current system. The Lsof tool is able to view this list for system detection and troubleshooting, common usage:

View file system blocking Lsof/boot

To see which process occupies the port number lsof-i: 3306

See which files users open lsof–u username

See which files the process opens Lsof–p 4838

View Remote Open Network links lsof–i @192.168.34.128

▲perf

Perf is a System performance optimization tool that comes with Linux kernel. The advantage is the close integration with the Linux kernel, which can be applied first to the new feature, which joins kernel, to view hotspot functions and to see the ratio of Cashe miss to help developers optimize program performance.

The basic principle of performance tuning tools such as perf,oprofile is to sample the monitored objects, and the simplest case is to sample from the tick interrupt, which triggers the sampling point within the tick interrupt, judging the context of the program at the sampling point. If a program spends 90% of its time on function foo (), then 90% of the sample points should fall into the context of function foo (). Luck is unpredictable, but I think that as long as the sampling frequency is high enough, the sampling time is long enough, then the above inference is more reliable. So, with tick-triggered sampling, we can see where in the program is the most time-consuming and focus on analysis.

For a deeper understanding of this tool, you can refer to:
http://blog.csdn.net/trochiluses/article/details/10261339

Summary: Combining the above-mentioned performance test commands and contacting the diagram of the Performance analysis tool at the beginning of the article, you can get a preliminary understanding of which aspects of performance analysis process to use which aspect of the tool (command).

"Common Performance testing Tools"

Proficient and proficient in the second part of the Performance Analysis command tool, introduced a number of performance testing tools, the introduction of the first simple to understand a few performance testing tools:

? Perf_events: A performance diagnostic tool that is distributed and maintained with Linux kernel code, maintained and developed by the kernel community. Perf can be used not only for performance statistical analysis of applications, but also for performance statistics and analysis of kernel code.

For more information: http://blog.sina.com.cn/s/blog_98822316010122ex.html.

? EBPF Tools: A tool for performance tracking using BCC, EBPF map can be used with custom EBPF programs that are widely used for kernel tuning, or to read user-level asynchronous code. It is important that this external data can be managed in user space. The map data body in this k-v format is managed by invoking the BPF system invocation creation, addition, deletion, etc. in user space. more:http://blog.csdn.net/ljy1988123/article/details/50444693.

? Perf-tools: A Linux performance analysis tuning toolset based on Perf_events (perf) and Ftrace. Perf-tools relies on fewer libraries and is easy to use. Supports Linux 3.2 and above kernel versions. More:https://github.com/brendangregg/perf-tools.

? BCC (BPF Compiler Collection): A Perf performance analysis tool that uses EBPF. A toolkit for creating efficient kernel tracking and operating procedures, including several useful tools and examples. Using the Extended BPF (Berkeley packet Filter), formally called EBPF, a new feature was first added to Linux 3.15. Multi-purpose requires more than 4.1 Bcc for Linux.

For more information: Https://github.com/iovisor/bcc#tools.

? Ktap: A new dynamic performance tracking tool for Linux scripts. Allows users to track Linux kernel dynamics. Ktap is designed to be interoperable, allowing users to tune operational insights, troubleshoot and extend the kernel and applications. It is similar to Linux and Solaris DTrace SystemTap. For more information: Https://github.com/ktap/ktap.

? Flame Graphs: is a graphical software that uses Perf,system Tap,ktap visualization, allowing the most frequent code paths to be quickly and accurately identified, and can be generated using a program developed in the Github.com/brendangregg/flamegraph source code.

For more information: http://www.brendangregg.com/flamegraphs.html.

First, Linux observability Tools | Linux Performance Observation Tools

? The basic tool to learn first is as follows:
Uptime, Top (htop), Mpstat, Isstat, Vmstat, free, ping, Nicstat, Dstat.

? The advanced commands are as follows:
SAR, netstat, pidstat, Strace, Tcpdump, Blktrace, Iotop, Slabtop, Sysctl,/proc.

For more information: http://www.open-open.com/lib/view/open1434589043973.html, the detailed command method can be used to refer to man

Second, Linux Benchmarking Tools | Linux Performance Assessment Tool

is a performance evaluation tool, for the performance test of different modules can use the corresponding tools, want to learn more, you can refer to the most relevant attachment document.

Third, Linux tuning Tools | Linux Performance Tuning Tool

is a performance tuning tool, mainly from the Linux kernel source layer of tuning, want to learn more, you can refer to the attached document below.

Iv. Linux observability SAR | Linux Performance observation Tools

SAR (System activity Reporter report) is one of the most comprehensive system performance analysis tools available on Linux, which can be used to report the activities of the systems in many ways, including: Read and write of files, usage of system calls, disk I/O, CPU efficiency, Memory usage, process activity, and IPC-related activities.

Common use of SAR: SAR [options] [-A] [-o file] t [n]
which
T is the sampling interval, n is the number of samples, the default value is 1;
-o file means that the command result is stored in a binary format in a file, and file is a filename.
Options is a command-line option

Ultra-complete finishing! Linux Performance analysis Tools Summary Collection