CentOS Service Program Performance Evaluation Document

1 Overview 1.1 factors affecting Linux Service Program Performance

CPU, memory, disk I/O bandwidth, network I/O bandwidth

1.2 Performance Evaluation

CPU: user % + sys % <70%; percentage of execution time of the program in user and kernel states.

Memory: Swap In (si) = 0; Swap Out (so) = 0; the Swap partition is not used. If Swap partition is frequently used, the memory may not be enough.

Hard Disk: iowait % <20%;

Network: as long as you have enough bandwidth to use it, achieving the NIC bandwidth in linux means no pressure.

Here: % user: indicates the percentage of time when the CPU is in user mode.

% Sys: the percentage of time when the CPU is in kernel mode.

% Iowait: Percentage of CPU waiting for input/output completion time.

Swap in: si, indicating page import of virtual memory, that is, switching from swap disk to RAM

Swap out: so indicates page export of the virtual memory, that is, switching from RAM to swap disk.

1.3 Performance Analysis Tools

Common system commands: top, free, ps, uptime, iotop, vmstat, iostat, dstat, and sar.
How to Use: use the top command to grasp the global conditions and use specific commands for in-depth analysis

Common combination methods:

(1) CPU bottlenecks: top, vmstat, iostat, sar-u, and sar-q

(2) memory bottlenecks: free, vmstat, sar-B, sar-r, and sar-W

(3) disk I/O bottlenecks: iotop, iostat, sar-B, sar-u, and sar-d

(4) Network bottleneck: dstat

2 top2.1 Functions

Provides real-time monitoring of the status of the system processor, memory, task, etc. This command can be used by CPU, memory usage to sort tasks; TOP is a dynamic display process, you can press the buttons to refresh the current status or specify the refresh interval at startup.

2.2 command output

Top-10:16:29 up 38 days, 5 users, load average: 0.04, 0.10, 0.05

Task: 569 total, 2 running, 562 sleeping, 0 stopped, 5 zombie

Cpu (s): 2.6% us, 1.3% sy, 0.4% ni, 95.7% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st

Mem: 3839112 k total, 3151560 k used, 687552 k free, 302944 k buffers

Swap: 6078456 k total, 631852 k used, 5446604 k free, 348548 k cached

Pid user pr ni virt res shr s % CPU % mem time + COMMAND

10603 zhixiang 20 0 1225 m 284 m 30 m S 4.6 7.6 271: 48.24 vs_exe

2473 xulun 20 0 118 m 43 m 6932 S 3.0 1.2: 964 Xvnc

1412 xulun 30 10 232 m 12 m 5216 S 2.3 0: 04. 59 floaters

14823 chujie 20 0 1112 m 247 m 17 m S 2.0 6.6 363: 51.38 vs_exe

17772 sihao 20 0 1101 m 202 m 10 m S 1.7 5.4 631: 21.17 vs_exe

11054 sihao 20 0 906 m 65 m 9412 S 1.0 1.7: 410 vs_exe

20782 yongtao 20 0 1049 m 173 m 9996 S 1.0 4.6: 196 vs_exe

14637 chujie 20 0 1274 m 132 m 2388 S 0.7 3.5 63: 20.18 KugooPopMsgServ

1439 yanyun 20 0 15428 1600 R 940 0.3 0: 00. 01 top

3491 zhixiang 20 0 129 m 55 m 17 m S 0.3. 25 Xvnc

1 root 20 0 19344 1200 S 984 0.0. 89 init

2.3 command output Parsing

Most of the output information includes all the performance indicators. The first five lines are the statistical information area, indicating the overall statistical information of the system, including: system load, task, CPU, memory, and so on; each process is followed by related information.

The first line is the task queue information, which is the same as the execution result of the uptime command.

[Yanyun @~ /Test] $ uptime

10:20:17 up 38 days, 5 users, load average: 0.00, 0.04, 0.03

The content is as follows: 10: 20: 17: Current Time

38 days, 15: 52: system running time

5 users: current number of logged-on users

Load average: 0.00, 0.04, 0.03: system load, that is, the average length of the task queue. The three values are the average values from 1 minute, 5 minutes, and 15 minutes ago to the present.

Information about the second-Action Process

Content: 569 total: total number of processes

2 running: Number of running Processes

562 sleeping: Number of sleeping Processes

0 stopped: Number of stopped Processes

5 zombie: Number of zombie Processes

Third behavior CPU Information

The content is as follows: 2.6% us: Percentage of CPU occupied by user space

1.3% sy: CPU usage of kernel space

0.4% ni: CPU usage percentage of processes that have changed their priorities in the process space

95.7% id: Percentage of idle CPU

0.0% wa: Percentage of CPU time waiting for Input and Output

The fourth and fifth lines indicate the memory information. The command output is the same as that of free.

[Yanyun @ ~] $ Free

Total used free shared buffers cached

Mem: 3839112 3256976 582136 0 143664 444992

-/+ Buffers/cache: 2668320 1170792

Swap: 6078456 574772 5503684

Content: Mem: 3839112 k total: total physical memory

3151560 k used: total physical memory used

687552 k free: Total idle memory

302944 k buffers: The amount of memory used as a buffer

Swap: 6078456 k total: total number of Swap Areas

631852 k used: the total number of swap areas used

5446604 k free: Total number of free swap Areas

348548 k cached: Total cache volume.

Note: buffer: it can be considered as the buffer for writing a disk;

Cache: Read the disk Cache.

In Linux, the principle of memory usage is: Do not use white, use it in white; cache as much as possible, so it often seems that the idle memory is small, but the cache is very large; the Linux system regularly starts the kernel thread kswapd for cache collection.

The information about each process is displayed.

% CPU: Percentage of CPU usage since the last update

TIME +: total cpu time used by the Process

% MEM: Percentage of physical memory used by the Process

VIRT: Total virtual memory used by the process, in kb

RES: The size of physical memory used by the process, not swapped out, in kb. RES = CODE + DATA

SHR: size of the shared memory, in kb

S: Process status. (D = non-disruptive sleep state R = running S = sleep T = tracking/stopping Z = botnets)

2.4 Common options

Top [-] [d] [p] [M] [P]

Parameter description:

D: Specifies the interval between two screen information refreshes. (Top-d 1: refresh every second)

P: Specifies the process ID to monitor only a process. (Top-d 1234: only view process information with pid 1234)

K: terminate a process. Top running parameters: the system prompts you to enter the PID of the process to be terminated and the signal to be sent to the process. Use signal 9 to force the process to end.

M: sort by resident memory size.

P: sort by CPU usage percentage.

Note: When executing the command line, press the number key '1' to view information about each CPU core.

Tasks: 564 total, 3 running, 556 sleeping, 0 stopped, 5 zombie

Cpu0: 2.9% us, 2.9% sy, 0.0% ni, 94.2% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st

Cpu1: 2.0% us, 2.0% sy, 0.0% ni, 96.1% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st

Cpu2: 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st

Cpu3: 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st

Mem: 3839112 k total, 3445296 k used, 393816 k free, 48180 k buffers

Swap: 6078456 k total, 553876 k used, 5524580 k free, 976128 k cached

Pid user pr ni virt res shr s % CPU % mem time + COMMAND

14823 chujie 20 0 1112 m 245 m 16 m R 3.0 6.5 373: 07.82 vs_exe

5589 sihao 20 0 1019 m 267 m 29 m R 2.0 7.1. 80 vs_exe

5674 zhixiang 20 0 1103 m 253 m 37 m S 2.0 6.8. 89 vs_exe

3 iotop: 3.1 features

Is a top tool used to monitor the disk I/O usage of each thread,

Note: You must install this command by yourself (yum install iotop)

3.2 command output

Total disk read: 50.23 M/s | Total disk write: 34.25 K/s

Tid prio user disk read disk write swapin io> COMMAND

61524 be/4 root 47.65 M/s 0.00 B/s 0.00% 37.83%./relay_server

61539 be/4 root 121.77 K/s 0.00 B/s 0.00% 26.73%./relay_server

61544 be/4 root 700.15 K/s 0.00 B/s 0.00% 24.89%./relay_server

61543 be/4 root 528.92 K/s 0.00 B/s 0.00% 21.29%./relay_server

61541 be/4 root 494.67 K/s 0.00 B/s 0.00% 21.22%./relay_server

61540 be/4 root 323.44 K/s 0.00 B/s 0.00% 8.62%./relay_server

61542 be/4 root 468.04 K/s 0.00 B/s 0.00% 8.13%./relay_server

480 be/3 root 0.00 B/s 0.00 B/s 0.00% 0.02% [jbd2/sda2-8]

1 be/4 root 0.00 B/s 0.00 B/s 0.00% 0.00% init

2 be/4 root 0.00 B/s 0.00 B/s 0.00% 0.00% [kthreadd]

3 rt/4 root 0.00 B/s 0.00 B/s 0.00% 0.00% [migration/0]

3.3 command output Parsing

The first line:

Total disk read: 50.23 M/s: volume of data READ from the DISK per second

Total disk write: 34.25 K/s: volume of data written to the DISK per second

The following is the IO situation of each thread:

It is relatively simple; just explain IO: it has the same meaning as top wa, but it only indicates the wa of a thread.

3.4 common options

Iotop [-] [d] [p]

Parameter description:

D: Specifies the interval between two screen information refreshes. (Iotop-d 1: refresh every second)

P: Specifies the thread ID to monitor only a thread. (Iotop-d 1234: only view process information with pid 1234)

Note: During command line execution, press 'O' to view only threads with IO.

Pstree-p: view the process tree and output the relationship between processes.

Ps-eLf: view the thread. Ps-ef advanced Edition

4 vmstat: 4.1 features

Tools used to view Virtual Memory usage

4.2 command output

[Yanyun @~ /Test] $ vmstat

Procs ----------- memory ---------- --- swap -- ----- io ---- system -- ----- cpu -----

R B swpd free buff cache si so bi bo in cs us sy id wa st

0 0 631376 834896 15108 452024 0 0 4 4 1 2 2 1 97 0

4.3 command output Parsing

Memory: the fourth and fifth rows of the same top.

Swap: swap-in/out rate from memory to swap partition.

Io: read/write disk Rate

System in: Number of interrupts per second, including clock interruptions

System cs: the number of environment (context) switches per second. frequent switches have an impact on the system. linux provides the cpu affinity for threads and can bind a thread to a single core for running.

Cpu: The third line of top output

Note: If the swap entry is not zero and all swap partitions are used, the memory usage may have become a bottleneck of the system.

4.4 common options

Vmstat 1 10: refresh once per second, refresh 10 times in total and exit.

Note: The following options are used to view the information of a specific item professionally. There are several items that output a large amount of information. Use google now!

-A: displays active and inactive memory.

-F: displays the number of forks since system startup.

-M: displays slabinfo

-S: displays memory-related statistics and the number of various system activities.

-D: Displays disk statistics.

-P: displays statistics of specified disk partitions.

5 iostat5.1 Function

Outputs CPU and disk I/O-related statistics

5.2 output results

[Root @ ShanWei_119_134_255_208 ~] # Iostat-x

Linux 2.6.32-279. el6.x86 _ 64 (ShanWei_119_134_255_208) 07/30/2013 _ x86_64 _ (16 CPU)

Avg-cpu: % user % nice % system % iowait % steal % idle

2.63 0.00 2.25 5.98 0.00 89.14

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm % util

Sda 195.42 163.72 194.33 126.28 28464.35 2319.94 96.02 1.16 3.63 1.41 45.30

Sdb 0.33 48.97 122.71 2.70 29603.37 413.38 239.34 0.27 2.15 1.00 12.52

5.3 command output explanation

Avg-cpu segment:

% User: the percentage of CPU used for running at the user level.

% Nice: the percentage of CPU used by the nice operation.

% Sys: Percentage of CPU used to run at the system level (kernel.

% Iowait: CPU usage when the CPU waits for hardware I/O.

% Idle: Percentage of idle CPU time.

Device segment:

Tps: Number of I/O requests sent per second

Blk_read/s: number of blocks read per second

Blk_wrtn/s: number of blocks written per second

Blk_read: Total number of read Blocks

Blk_wrtn: Total number of blocks written

-X option to view more information

Rrqm/s: the number of Merge read requests related to the device per second

Wrqm/s: the number of Merge write requests related to this device per second

Rsec/s: number of sectors read per second

Wsec/: number of sectors written per second

Avgrq-sz: average data size (sector) of each device I/O operation)

Avgqu-sz: Average I/O Queue Length

Await: average wait time for each device I/O operation (unit: milliseconds)

Svctm: Average service time for each device I/O operation (unit: milliseconds)

% Util: All I/O processing time in the statistical time, so this parameter implies the degree of Device Busy.

5.4 common options

Iostat-d 2 6: displays device statistics every two seconds. A total of six output times are displayed.

-X: view the device usage and response time.

6 dstat6.1 Functions

It is an all-around system information statistics tool that can only monitor the entire system, rather than in-depth analysis of a process or a program. The monitoring items include: cpu, disk, memory, network card, process, system (color interface is rare in linux !)

Note: This Command needs to be installed by yourself (yum install dstat)

6.2 command output

[Root @ ShanWei_119_134_255_208 ~] # Dstat-cdlmnpy

---- Total-cpu-usage -----dsk/total---- load-avg --- ------ memory-usage ------net/total---- procs --- system --

Usr sys idl wai hiq siq | read writ | 1 m 5 m 15 m | used buff cach free | recv send | run blk new | int csw

3 2 89 6 0 0 | 28 M 1367k | 5.41 5.07 4.87 | 4019 M 172 M 58.5G 267M | 0 0 | 0 0.0 1.3 | 8608 35 k

2 1 85 12 0 0 | 21 M 828k | 5.41 5.07 4.87 | 4020 M 172 M 58.6G 243M | 191 k 15 M | 0 6.0 0 | 5711 18 k

1 1 80 18 0 0 | 5828 k 4804k | 5.41 5.07 4.87 | 4018 M 172 M 58.6G 260M | 190 k 17M | 2.0 3.0 5802 0 | 12 k

6.3 command output explanation

I have explained all of the above ......

6.4 common options

Dstat-cdlmnpsy: The options cdlmnpsy include common ones. Of course, there are a lot of dstat -- help.

7. sar7.1 Functions

The System Activity Reporter System Activity report is one of the most comprehensive System performance analysis tools on Linux. It can report System activities from multiple parties, including: file read/write, system call usage, disk I/O, CPU efficiency, memory usage, process activity, and IPC-related activity.

7.2 command output

[Root @ localhost ~] # Sar-r 1 100

Linux 2.6.32-220. el6.x86 _ 64 (localhost. localdomain) 07/30/2013 _ x86_64 _ (8 CPU)

03:17:59 kbmemfree kbmemused % memused kbbuffers kbcached kbcommit % commit

03:18:00 4988488 3062224 38.04 639136 2115404 72404

03:18:01 4984464 3066248 38.09 639136 2115404 98060

03:18:02 4985152 3065560 38.08 639136 2115420 97972

03:18:03 4985400 3065312 38.08 639136 2115420 97972

[Root @ localhost ~] # Sar-B 1 100

Linux 2.6.32-220. el6.x86 _ 64 (localhost. localdomain) 07/30/2013 _ x86_64 _ (8 CPU)

03:19:09 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s % vmeff

03:19:10 0.00 157.14 45415.48 0.00 15541.67 0.00 0.00 0.00 0.00

03:19:11 0.00 0.00 40.40 0.00 75.76 0.00 0.00 0.00 0.00

03:19:12 0.00 0.00 5542.34 0.00 1584.68 0.00 0.00 0.00 0.00

[Root @ localhost ~] # Sar-B 1 100

Linux 2.6.32-220. el6.x86 _ 64 (localhost. localdomain) 07/30/2013 _ x86_64 _ (8 CPU)

03:19:38 tps rtps wtps bread/s bwrtn/s

03:19:39 PM 12.50 0.00 12.50 0.00 181.82

03:19:40 PM 0.00 0.00 0.00 0.00 0.00

03:19:41 PM 7.14 0.00 7.14 0.00 128.57

[Root @ localhost ~] # Sar-W 1 100

Linux 2.6.32-220. el6.x86 _ 64 (localhost. localdomain) 07/30/2013 _ x86_64 _ (8 CPU)

03:20:10 pswpin/s pswpout/s

03:20:11 PM 0.00 0.00

03:20:12 PM 0.00 0.00

03:20:13 PM 0.00 0.00

[Root @ localhost ~] # Sar-d 1 100

Linux 2.6.32-220. el6.x86 _ 64 (localhost. localdomain) 07/30/2013 _ x86_64 _ (8 CPU)

03:20:48 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm % util

03:20:49 dev8-0 7.69 0.00 79.12 10.29 0.00 0.14 0.14 0.11

03:20:49 dev8-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

03:20:49 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm % util

03:20:50 dev8-0 10.47 0.00 148.84 14.22 0.08 7.22 7.11 7.44

03:20:50 dev8-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

7.3 command output explanation

Kbmemfree: this value is basically the same as the free value in the free command, so it does not include the buffer and cache space.

Kbmemused: this value is basically the same as the used value in the free command, so it includes the buffer and cache space.

% Memused: this value is a percentage of kbmemused and total memory (excluding swap.

Kbbuffers and kbcached: These two values are buffer and cache in the free command.

Kbcommit: Ensure the memory required by the current system, that is, the memory required to ensure no overflow (RAM + swap ).

% Commit: this value is a percentage of kbcommit and total memory (including swap.

Pgpgin/s: the number of bytes per second from disk or SWAP to memory (KB)

Pgpgout/s: the number of bytes per second from memory replacement to disk or SWAP (KB)

Fault/s: number of missing pages generated by the system per second, that is, the sum of the Primary and Secondary missing pages (major + minor)

Majflt/s: Number of Primary missing pages per second.

Pgfree/s: Number of pages per second in the idle queue

Pgscank/s: Number of pages scanned by kswapd per second

Pgscand/s: number of directly scanned pages per second

Pgsteal/s: the number of pages that are cleared from the cache every second to meet memory requirements

% Vmeff: Percentage of pages cleared per second (pgsteal) to total scanning pages (pgscank + pgscand)

Tps: the total amount of I/O transferred by physical devices per second

Rtps: Total amount of data read from physical devices per second

Wtps: Total amount of data written to the physical device per second

Bread/s: the amount of data read from the physical device per second. Unit: block/s.

Bwrtn/s: the volume of data written to the physical device per second. Unit: block/s.

Pswpin/s: Number of swap pages per second

Pswpout/s: Number of swap pages generated by the system per second

7.4 common options

Sar-[u] [r] [B] [B] [q] [W] [d]

Parameter description:

-U: CPU resource monitoring

-R: memory and swap space monitoring

-B: Memory paging monitoring

-B: IO and transmission rate monitoring

-Q: Process queue system load monitoring

-W: System swap partition activity monitoring

-D: Device usage monitoring

8 tmpfs8.1 Definition

Tmpfs is a memory-based file system mounted under/dev/shm.

Features 8.2

Dynamically change the size. The access is fast and completely resident in RAM. Of course, the system will not be restarted.

8.3 usage

1. POSIX standard shared memory in linux is implemented based on this file System (there is also a set of System V standard shared memory implementation methods );

2. If you create a file under/dev/shm, tmpfs will be used directly. If you do not need to consider the loss of data after the machine restarts, put the data below, release your disk!

3. The default size of tmpfs is half of the physical memory. Adjust the maximum capacity to 40 GB and the number of file nodes to 1000000. Run the following command:

Mount-o size = 40G-o nr_inodes = 1000000-o noatime, nodiratime-o remount/dev/shm

4. If you need to permanently modify the size of the tmpfs file system, you need to modify/etc/fstab.

9 crontab: 9.1 Function

Scheduled execution of some commands, which is equivalent to a system-level timer. In/var/spool/cron/, there are some files named after the user name, indicating that the user's scheduled task is executed.

9.2 configuration method

Format description:

* ***/Command path

The first five fields can take an integer to specify when to start working (minutes, hours, days, months, and weeks), the sixth field is a string, scripts executed upon expiration, programs, etc.

43 21 *** run at every day

0 17 ** 1 run at every Monday

17 **, 3 run at and on every Sunday, Tuesday, and Wednesday.

0-10 17 1 ** every 1 minute from to on the first day of the month

0 0*1 on the 1st and 15th of the month

42 4 1 ** executed at on the first day of the month

0 21 ** 1-6 from Monday to Saturday

0, 10, 20, 30, 40, 50 *** run every 10 minutes

*/10 * execute every 10 minutes

* 1 *** run every 1 minute from

0 1 *** execution

0 */1 * execution at every 1 hour

0 * execution at every 1 hour

2 8-20/3 *** run

30 5, ** executed at on the 1st and 15th

The following configuration indicates that the script/opt/clear_old_file.sh is executed at every day;

This script function is used to delete files not accessed within five days under the/data1/data2 directories.

0 3 *** (cd/opt/&./clear_old_file.sh>/dev/null &)

[Root @ ShanWei_119_134_255_208 ~] # Cat/opt/clear_old_file.sh

#! /Bin/sh

Find/data1/*-type f-atime + 5-exec rm {}\;

Find/data2/*-type f-atime + 5-exec rm {}\;

Exit 0

10 proc: To be continued...

The/proc file system is unique to GNU/Linux. It is a virtual file system, which completely resides in RAM, so all files in this directory do not consume disk space. It can be used to easily understand kernel information and hardware information in the system. It can also be used to configure system kernel parameters. Many Commands actually collect information from/proc files, and then organize and display the information according to their own format. This is basically the same as the commands described above.