Introduction to Linux Performance tools

L Linux Performance Tools Introduction

P CPU High

P Disk I/O

P Network

P Memory

P Application Tracking

L The relationship between the operating system and the application is likened to "Chunwangchihan".

Application performance issues/functional issues, in addition to the use of online debugging, logging, the operating system provides a wealth of tools to enable you to perspective the application, problem location analysis more efficient

I introduced a lot of Linux tool usage data, this does not introduce tools to use, but tells the tool behind the meaning of the numbers, as well as our usual common misunderstanding of the tool output

CPU Hi-uptime

L Uptime and top commands show the last 1 minutes, 5 minutes, 15 minutes of load

All the processes in Linux are placed in a queue called the run queue, the above output represents the number of processes that are already running (get to the CPU) + waiting to run, for example, 0.62 above indicates the number of queues that were running in the last 1 minutes: 0.62*60 = 37

L only have 1 CPUs, but when multicore, it is similar to multi-lane, assuming the n core, the maximum acceptable load is n.0. View/proc/cpuinfo in Linux get the number of CPUs

L This value is generally more than twice times the CPU core indicates a serious performance problem, such as the assumption of a 8-core computer, this value continues to more than 16, stating that the wait is serious, but if only a moment, you can not pay too much attention

L Uptime and top can only see the last 1, 5, 15 minutes,

CPU Hi-vmstat

If you want to see the real-time queue length, view the output of the Vmstat command in the first column: R

L The output of the Vmstat command is informative and only describes CPU-related metrics

L R: The length of the run queue mentioned in the previous uptime command is the same, if this number is greater than twice times the CPU core number indicates a serious performance bottleneck

L B: The number of processes in the uninterruptible sleep state. There are 2 ways that a process goes to sleep, interruptible sleep: An acceptable signal or a wake-up, uninterruptible sleep means no reception, Can only be awakened to the corresponding event, generally entering this state is to wait for I/O (disk I/O or network I/O, need to be judged separately). Similarly, if you are in B, it can mean I/O has a bottleneck.

L US/SY/ID/WA:

L US: The amount of time spent in a non-nuclear mindset, this if high generally means that the application's algorithm implementation has performance issues

L SY: The time taken in the nuclear mentality, generally refers to the time spent by the system call

L ID: Idle time

L WA: Time spent waiting on Io, note that this indicator is somewhat difficult to understand, on the next page will be explained in detail

L wa:iowait Nearly 100% does not mean that the system is unhealthy, the same reason iowait 0% is not meant to be a healthy system. Because we know that the performance of the CPU is increased by one times every 12 or 18 months, and the performance of the disk is limited by the physical characteristics of its IOPS (IO Operations per Second), while iowait = (CPU is idle and waiting for IO to finish)/(CPU runtime + (CPU idle and waiting for IO completion time)

L Suppose a CPU, because of poor CPU performance, the last calculated%iowait is 33%.

CPU time = ms
IO time = ms
Total transaction time = CPU + IO = + + =
%iowait = IO time/total time = 20/60 = 33%

L then changed to another CPU, this CPU performance has greatly improved, CPU computation time greatly reduced:

CPU time = MS/4 = Ten ms
IO time = ms
Total transaction time = CPU + IO = ten + ms
%iowait = 20/30 = 66%

I can see from the above that CPU performance has improved, but%iowait has increased by twice times.

L iowait is an indication of I/O may be problematic and needs to be further combined with other data analysis

L iowait comparison between different hardware doesn't make much sense.

CPU High- See which process CPU High

The 6th column in the L PS H-eo user,pid,ppid,tid,time,%cpu,cmd--sort=%cpu is a small to large list of CPU consumption

After the process is found, the stack is obtained based on the type of process being parsed:

P If it is a Java program, if the IBM JDK uses kill-3 to generate Java core, if it is a sun jdk, use the JDK's own program Jstack build

P If the program is compiled in C + + or C, use gstack<pid> to generate a stack of threads

When analyzing the stack, if the CPU is high, you need to get the stack to compare multiple times, exclude threads in wait, sleep, etc., if a thread persists and is not waiting, it could be the culprit.

disk I/O Hi-iostat

L mentioned Vmstat in the previous page%iowait to prepare to determine if there is an IO bottleneck is not exactly what the better indicator is IOPS, that is, the output of the Iostat TPS column:

L IOPS is a relatively professional term,

The maximum iops,iops =1000ms/(tseek+trotation) is calculated based on the different disk types, so that the maximum iops of the disk, which is ioPS = 1000ms/(Tseek + troatation), can theoretically be calculated. Data transfer time is ignored. Assuming that the average disk seek time is 3MS, the disk speed is 7200,10k,15krpm,trotation generally for disk rotation of 1/2 calculation, the maximum disk ioPS theoretical value, respectively,

IOPS = 1000/(3 + 60000/7200/2) = 140

IOPS = 1000/(3 + 60000/10000/2) = 167

IOPS = 1000/(3 + 60000/15000/2) = 200

disk I/O High

In most cases, we have not reached the disk I/O bottleneck, more often our application needs to be optimized, the front iowait or vmstat b column can only see the specific I/O, such as to see which process I/O is high, there are 2 ways:

P third-party tools Iotop, http://guichaz.free.fr/iotop/

P or relatively simple: for x in ' SEQ 1 1 10 '; Dops-eo State,pid,cmd | grep "^d"; echo "----"; Sleep 5;done

This command means to check that the status is in D (that is, the process in front of the uninterruptable state, typically waiting for I/O)

Network problems

L traceroute is used to detect if the route to the destination is correct, and sometimes some routes are incorrectly configured to cause the message to go the wrong way.

L Ping: This goes without saying.

L sar-n dev-i All <interval> <count> detects traffic for each NIC. For large-volume applications, such as portals, it is possible to fully fill the traffic. Don't think 100M's website actually really can reach 100M bits , is limited by the actual ability of network card or network cable, and can not reach true 100M bits

L Tcpdump grasping the package must be, note if the machine is native to this machine, the network card has to be lo (loop back)

L netstat View the status of the socket. For example: time_wait status is often present. Network connections can not be established, such as the State may also be analyzed by netstat combined with tcpdump, for example: DNS/etc/resolve.conf configuration is incorrect, resulting in gethostbyname from the domain name to obtain the IP, will be queried multiple times

High memory consumption

L page Cache (Cache&buffer) in order to cache data on disk as much as possible for memory-boosting performance use, page cache includes 2 parts: The file's data block and the file's metadata (Supberblock, inodes, bitmaps), such as the free, top and other commands to display the front as cached, the latter is shown as buffer, these 2 and is the page cache

Free is not the real idle memory of Linux, the buffer and the cache are added

High memory consumption

When memory is low, slab, buffer, and cache will first be thin, constantly shrink, and start using swap, when using Vmstat will see a lot of swap in and out (SI and so)

L IF the Vmstat si and so data is large, indicating that the memory is not enough, use PS aux to see which process memory consumption, the PS aux output to see the RSS (Resident memory) column

For processes such as Java, do not use the Linux operating system commands to view, but instead use the JVM's own tools, such as: Jmap, because these tools are self-managing memory

Other

L STRACE The system call of the trace process, for example: A program in the process of running suddenly hang, you can use Strace trace call in which system call hangs

L GDB Debugging Application

L/proc file system, you can see almost all the status of the process, such as locating the socket hook death, interested in the/usr/src/linux/documentation/filesystems/proc.txt of the single Board to study

Introduction to Linux performance Tools