Advanced Performance Test Guide-Basic one (system resources explained)

This article is intended to help testers to provide a simple explanation of the performance test common indicators, mainly including CPU, memory, disk and network bandwidth and other system resources, this article is confined to the Linux system, the Windows server system is not considered.

I. Analysis of system resources

Linux system resources are mainly recorded in the kernel file/proc, the following almost records all the Linux system information.

1. CPU1.1 CPU File System

The basic information of the CPU can be viewed under directory cpuinfo, unlike the/proc/cpuinfo files generated by CPUs based on different instruction sets (ISA), the/proc/cpuinfo file based on the X86 instruction set CPU contains the following:

[[email protected] proc]# cat cpuinfoprocessor       : 0vendor_id       : GenuineIntelcpu family      : 6model           : 45model name      :        Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHzstepping        : 7cpu MHz         : 2400.000cache size      : 10240 KBphysical id     : 0siblings        : 4core id         : 0cpu cores       : 4apicid          : 0fpu             : yesfpu_exception   : yescpuid level     : 13wp              : yesflags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc nonstop_tsc pni cx16 popcnt lahf_lmbogomips        : 4800.00clflush size    : 64cache_alignment : 64address sizes   : 40 bits physical, 48 bits virtualpower management: [8]

The above output items have the following meanings:

• Processor: The number of the logical processing cores in the system. For a single-core processor, it can be considered its CPU number, and for multicore processors it can be a physical core, or a virtual logical core using Hyper-Threading Technology
• VENDOR_ID:CPU Manufacturers
• CPU FAMILY:CPU Product Family Code
• Model:cpu belongs to which generation in its series
• Model NAME:CPU belongs to the name and its number, the nominal frequency
• Stepping:cpu belongs to the production update version
• CPU MHZ:CPU The actual use of the main frequency
• Cache Size:cpu Level Two buffer size
• Physical ID: The label of a single CPU
• Siblings: Number of logical physical cores for a single CPU
• Core ID: The number of the current physical core in which the CPU is located, which is not necessarily sequential
• CPU Cores: The physical core number of the CPU at which the logical core is located
• Apicid: Used to distinguish the number of different logical cores, this number of each logical core in the system must be different, this number is not necessarily continuous
• FPU: Whether it has a floating point unit (floating)
• Fpu_exception: Whether floating point calculation exceptions are supported
• CPUID level: The value in the EAX register before the CPUID instruction is executed, depending on the value of the CPUID instruction will return different content
• WP: Indicates whether the current CPU supports write protection of user space in kernel state (write Protection)
• Flags: Features currently supported by the CPU
• Bogomips: Approximate CPU speed at boot of system kernel (Million instructions Per Second)
• Clflush size: per-flush cache
• Cache_alignment: Cache Address Alignment Units
• Address sizes: Number of accessible addresses space
• Power Management: Support for energy management with the following optional support features:

Based on the above, we can easily know the current system about the CPU, CPU cores, CPU is enabled Hyper-threading and other information.

• How many logical cores a query system has :cat /proc/cpuinfo | grep "processor" | wc -l

• to query the physical core number of the system CPU :cat /proc/cpuinfo | grep "cpu cores" | uniq

• To query whether the system CPU is Hyper-Threading enabled :cat /proc/cpuinfo | grep -e "cpu cores" -e "siblings" | sort | uniq

  Output Example:
  cpu cores : 4
siblings : 4

If the number of CPUs cores and the number of siblings are the same, hyper-threading is not enabled, otherwise Hyper-threading is enabled.

• number of query system CPUs:Cat/proc/cpuinfo | grep "Physical ID" | Sort | Uniq | Wc-l

Query whether the system CPU supports a function, then the root is similar, the output of the sort, uniq and grep can get the results.

Calculation of 1.2 CPU

A top command

* First line:
09:27:32 当前系统时间
18 days, 1:02 系统已经运行了18天1小时2分钟（在这期间没有重启过）
5 users 当前有5个用户登录系统
load average: 10.19, 15.27, 18.08 load average后面的三个数分别是1分钟、5分钟、15分钟的负载情况。
The load average data is the number of active processes that are checked every 5 seconds and then calculated by a particular algorithm. If this number is divided by the number of logical CPUs, the result above 5 indicates that the system is overloaded.

* Second line:
Tasks 任务（进程），系统现在共有183个进程，其中处于运行中的有1个，182个在休眠（sleep），stoped状态的有0个，zombie状态（僵尸）的有0个。

* Third line: CPU status
60.48% us 用户空间占用CPU的百分比。
13.5% sy 内核空间占用CPU的百分比。
0.0% ni 改变过优先级的进程占用CPU的百分比
19.0% id 空闲CPU百分比
2.5% wa IO等待占用CPU的百分比
0.3% hi 硬中断（Hardware IRQ）占用CPU的百分比
4.3% si 软中断（Software Interrupts）占用CPU的百分比
In this case the CPU usage ratio is different from the Windows concept, and if you don't understand the user space and kernel space, it needs to be recharged.

* Line fourth: Memory status
48617324k total 物理内存总量（46GB）
46900564k used 使用中的内存总量（44.7GB）
1716760k free 空闲内存总量（1.6G）
2432120k buffers 缓存的内存量 （2.3G）

* Line fifth: Swap swap partition 34996216k total 交换区总量（33GB） 128k used 使用的交换区总量（128k） 34996088k free 空闲交换区总量（33GB）11883392k cached 缓冲的交换区总量（11GB）

The explanation here is that this data cannot be understood in terms of the memory concept of Windows. The memory management of Linux has its particularity, the complex point needs a book to explain, here is simply to say the point and our traditional concept (Windows) is different.

The total amount of memory in use in row four (used) refers to the amount of memory that is now controlled by the system kernel, and the total amount of free memory that the kernel has not included in its control range. The memory that is included in kernel management is not always in use, but also includes the memory that has been used in the past that can now be reused, and the kernel does not return these reusable memory to free, so there is less memory on Linux, but don't worry about it.
If you are accustomed to calculating the number of available memory, here is an approximate formula: The fourth line of Free + fourth row of buffers + fifth row of cached, press this formula for this server's available memory: 1716760+2432120+11883392 = 15.3GB.

For memory monitoring, in the top we have to monitor the fifth line swap partition used, if this value is constantly changing, indicating that the kernel is constantly in memory and swap data exchange, which is really not enough memory.

* Line sixth is blank

* Line seventh below: status monitoring of each process (Task)
* PID Process ID
* USER Process Owner
* PR Process Priority
* NI nice value. Negative values indicate high priority, positive values indicate low priority
* The total amount of virtual memory used by the VIRT process, in kilobytes. Virt=swap+res
* The amount of physical memory, in kilobytes, used by the RES process and not swapped out. Res=code+data
* SHR shared memory size, in kilobytes
* S process status. d= non-disruptive sleep status r= run s= sleep t= track/stop z= zombie Process
*%cpu CPU time consumption percentage last updated to current
* Percentage of physical memory used by the%MEM process
* Total CPU time used by time+ process, Unit 1/100 sec
* Command process name (commands name/command line)

Multi-u multi-core CPU monitoring

In the top basic view, press the keypad number 1 to monitor the status of each logical CPU:

Observe that the server has 8 logical CPUs, which are actually 2 physical CPUs.

Process Field Sort

By default, when Top is reached, the processes are sorted according to CPU consumption, with the process ID 14613 of the Java process First (CPU 390.7%), and the process ID 27924 Java process second (CPU occupied 7%). You can change the sort field by keyboard instructions, for example, to monitor which process consumes the most mem, I use the following method:

1. Tap Keyboard B (turn on/off the highlight effect), and the top view changes as follows:

We find that the top process with process ID 16899 is highlighted, and the top process is the one that is the only running state (runing) shown in the second row of the view, and you can turn off or turn on the highlight effect of the running process by tapping the Y key.

As you can see, the top default sort column is%cpu.

With SHIFT + > or SHIFT + <, you can change the sort order to the right or left by pressing SHIFT + > once:

The view is now sorted according to%mem.

Change the Process display field

1. Tap the F key and top into another view, where you can arrange the display fields in the base view:

Here is a list of all the process fields that can be displayed in the top Basic view, fields with * and uppercase letters are visible, fields without * and lowercase are not displayed. If you want to display the code and data two fields in the basic view, you can tap the R and S keys:

Return to Basic view, you can see more code and data two fields:

Additions to the top command

The top command is the preferred command for system monitoring on Linux, but sometimes it does not reach our requirements, such as the current server, where top monitoring has a lot of limitations. The minimal unit of monitoring for the top command is the process, so there is no concern about the number of Java threads and the number of client connections, and these two metrics are very important metrics for Java Web Services, usually with PS and netstate two commands to complement top deficiencies.

To monitor the number of Java threads:
ps -eLf | grep java | wc -l

Monitor network client connections:
netstat -n | grep tcp | grep 侦听端口 | wc -l

The above two commands, can change the parameters of grep, to achieve more detailed monitoring requirements.

In the Linux system everything is the idea of the implementation of the guidance, all processes running state can be used to obtain files. system root/proc, each digital subdirectory name is the PID of the running process, into any process directory, through which files or directories to observe the process of the various operational indicators, such as the task directory is used to describe the thread of the process, Therefore, you can also get the number of threads running in a process by using the following method (PID refers to the process ID):
ls /proc/PID/task | wc -l

There is also a command pmap in Linux to output the state of the process memory that can be used to analyze the thread stack:
pmap PID

b Calculation of CPU usage

/proc/stat file

The file contains information about all CPU activity, and all values in the file are accumulated from the start of the system to the current moment. The format of the file may be inconsistent in different kernel versions, and the following examples illustrate the meaning of each field in the data file. Instance data: On the 2.6.18-164.EL5 version

CPU 354452949 16260 60331937 380649864 4361825 1248983 9367391 0cpu0 38204143 1070 7122259 53682412 638525 46222 141618 0 CPU1 42667515 2209 7720403 48255130 605091 0 106222 0cpu2 46253571 1199 8501463 43893118 588902 106925 0cpu3 47755506 9 35 8960373 41907018 579979 734 109005 0cpu4 37011661 2915 7258339 51363615 472962 327902 5318893 0cpu5 46282988 3828 72164 49389385 597243 0 126783 0cpu6 49721915 2594 7962514 45251453 559504 167 128630 0cpu7 46555646 1506 5590168 46907728 31  9617 873936 3329313 0intr 2347835587 1023833604 9 0 3 3 0 5 0 1 0 0 0 115 0 13945415 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 142398948 0 0 0 0 0 0 0 1167657484 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0ctxt 5406149033btime 1398039911processes 2852119procs_running 5procs_blocked 0 

The number on the first line represents the total CPU usage, so we just use the first line of numbers to calculate it. The following table resolves the meanings of the first row of values:

Parameter Resolution (unit: Jiffies)

(Jiffies is a global variable in the kernel, used to record the number of Beats generated from the start of the system, in Linux, a beat is roughly understood as the minimum time slice of the operating system process scheduling, different Linux cores may have different values, usually between 1ms to 10ms)

• User (354452949) accumulates from the start of the system to the current time, in the runtime of the client state, and does not contain the nice value as a negative process.

• Nice (16260) accumulate from the start of the system to the current moment, the CPU time occupied by a process with a negative nice value

• System (60331937) accumulates at the current moment from the start of the systems and is in the running time of the nuclear mentality

• Idle (380649864) accumulates from the start of the system to the current moment, waiting time other than IO wait time iowait (12256) accumulate from the start of the system to the current moment, IO wait time (since 2.5.41)

• IRQ (4361825) accumulates from the start of the system to the current moment, hard interrupt time (since 2.6.0-TEST4)

• SOFTIRQ (1248983) accumulates from the start of the system to the current moment, soft interrupt time (since 2.6.0-TEST4) Stealstolen (0) which is the the-the-times spent in other operating systems When running in a virtualized environment (since 2.6.11)

• Guest (9367391) which is the time spent running a virtual CPU for guest operating systems under the control of the Linux Ke Rnel (since 2.6.24)

Conclusion 1: Total CPU Time Totalcputime = user + nice + system + Idle + iowait + IRQ + SOFTIRQ + Stealstolen + Guest

/proc/<pid>/stat file

The file contains information about all the activities of a process, and all values in the file are accumulated from the start of the system to the current moment. The following example shows the meaning of each field in the file using the instance data.

[[email protected] lib]# cat /proc/20738/stat20738 (java) S 1 20730 20615 34827 20615 4202496 13527950 0 2442 0 2593008 426148 0 0 22 0 1405 0 156378640 60392562688 6497342 18446744073709551615 4194304 4196484 140737061833664 18446744073709551615 243481475045 0 0 2 16800973 18446744073709551615 0 0 17 5 0 0 0

Description: The following only explains the relevant parameters that are useful for calculating CPU usage

Parameter interpretation

1--pid=20738 Process Number

The name of the 2--comm= (Java) application or command

Status of the 3--task_state=s task: R is running, S-is sleeping, D was sleeping in a uninterruptible wait, Z is zombie, T is traced or S Topped

4--ppid=1 Parent Process ID

5--pgid=20730 Thread Group number

6--SID=20615 the session group ID where the task resides

7--tty_nr=34827 the device number of the TTY terminal for the task, INT (34817/256) = main device number, (34817-Main device number) = Secondary device number

The process group number of the 8--tty_pgrp=20615 terminal, the PID of the foreground task (including the shell application) currently running on the terminal where the task is located.

9--task->flags=4202496 the process flag bit to see the properties of the task

10--min_flt=13527950 the number of page faults that occur when the task does not need to copy data from the hard disk

11--cmin_flt=0 cumulative number of times that all waited-for processes of the task have occurred

12--maj_flt=2442 the number of missing pages (main pages) that the task requires to copy data from the hard disk

13--cmaj_flt=0 cumulative number of main pages that have occurred in all waited-for processes of the task

14--utime=2593008 the time that the task was run in the user state, in units of jiffies

15--stime=426148 the mission at the time of the nuclear mentality run, the unit is jiffies

16--cutime=0 cumulative of all the waited-for processes of the task once in the user state run time, in units of jiffies

17--cstime=0 cumulative of all the waited-for processes of this task ever run in the nuclear mentality at the time, in units of jiffies

18--priority=22 the dynamic priority of a task

19--nice=0 the static priority of a task

20--num_threads=1405 the number of threads in the thread group where the task resides

21--it_real_value=0 the delay of the next SIGALRM send process due to the timing interval, in Jiffy.

22--start_time=5882654 the time the task was started, in Jiffies

23--vsize=1409024 (page) The virtual address space size of the task

24--RSS=56 (page) the size of the physical address space that the task currently resides in. Number of pages The process has in real Memory,minu 3 for administrative purpose. These pages may be used for code, data, and stacks.

25--rlim=4294967295 (bytes) The maximum value that the task can reside in the physical address space

26--start_code=134512640 the start address of the code snippet for the task in the virtual address space

27--end_code=134513720 the end address of the code snippet for the task in the virtual address space

28--start_stack=3215579040 the end address of the stack for the task in the virtual address space

29--kstkesp=0 the current value of the ESP (32-bit stack pointer), consistent with the kernel stack page of the process.

30--kstkeip=2097798 the current value of the EIP (32-bit instruction pointer) that points to the pointer that will be executed.

31--pendingsig=0 the bitmap of the pending signal, recording the normal signal sent to the process

32--block_sig=0 a bitmap for blocking signals

33--sigign=0 the bitmap of the ignored signal

34--sigcatch=082985 the bitmap of the captured signal

35--wchan=0 If the process is a sleep state, this value gives the scheduled call point

Nswap is swapped the number of pages, currently useless

Cnswap of the number of pages that are swapped by all child processes, currently useless

Exit_signal=17 the signal sent to the parent process at the end of the process

Task_cpu (Task) =0 on which CPU to run

Task_rt_priority=0 relative priority levels for real-time processes

task_policy=0 process scheduling strategy, 0 = non-real-time process, 1=FIFO real-time process; 2=RR real-time process

Conclusion 2: The total CPU time of the process processcputime = utime + stime + cutime + cstime, which includes the CPU time of all its threads.

Basic idea of calculating CPU utilization rate in single-core case

By reading/proc/stat,/proc//stat, and/proc/cpuinfo, the files get the total CPU time, the CPU time of the process, and the number of CPUs, and then calculate by some algorithm ( Sample two short enough time intervals for CPU snapshots and process snapshots to calculate CPU utilization of the process.

Total CPU Usage Calculation

Calculation method:

1, sampling two short enough time interval CPU snapshot, respectively recorded as T1,T2, wherein the structure of T1, T2 are: (User, nice, system, idle, iowait, IRQ, SOFTIRQ, Stealstolen, guest) 9-tuple;

2, calculate the total CPU time slice Totalcputime

A) sum up all CPU usage for the first time and get S1;

b) sum up all CPU usage for the second time, get S2;

c) S2–s1 get all time slices within this time interval, i.e. totalcputime = j2–j1;

3, Calculate idle time idle

Idle corresponds to the fourth column of data, with the second fourth column-the first fourth column can be

idle= Second fourth column-fourth column of the first time

6. Calculate CPU Usage

Pcpu =100* (Total-idle)/total

Calculation of CPU utilization for a process

Calculation method:

1. Sample two short enough time intervals for the CPU snapshot with the process snapshot,

A) Each CPU snapshot is a 9-tuple (user, nice, system, idle, iowait, IRQ, SOFTIRQ, Stealstolen, guest);

b) Each process snapshot is a 4-tuple (Utime, stime, Cutime, cstime);

2. According to conclusion 1, conclusion 2 calculates the total CPU time and process CPU time of two time, respectively: TotalCpuTime1, TotalCpuTime2, processCpuTime1, processCpuTime2

3. Calculates the CPU utilization of the process PCPU = 100* (processcputime2–processcputime1)/(TOTALCPUTIME2–TOTALCPUTIME1) (calculated by 100%, If it is multi-core, multiply the number of CPUs);

Posted by Gaochuanjun 9th, octopress

Advanced Performance Test Guide-Basic one (system resources explained)