Linux server performance Analysis and tuning

A Linux server Performance View 1.1 CPU Performance View 1, view the number of physical CPUs:

cat /proc/cpuinfo |grep "physical id"|sort|uniq|wc -l

2. View the number of cores in each physical CPU:

cat /proc/cpuinfo |grep "cpu cores"|wc -l

3, the number of logical CPUs:

cat /proc/cpuinfo |grep "processor"|wc -l

Number of physical CPUs * Number of cores = number of logical CPUs (in case of Hyper-Threading technology not supported)

1.2 Memory View 1, view memory usage:

#free -m             total       used       free     shared    buffers     cachedMem:          3949       2519       1430          0        189       1619-/+ buffers/cache:        710       3239Swap:         3576          0       3576total：内存总数used：已经使用的内存数free：空闲内存数shared：多个进程共享的内存总额- buffers/cache：(已用)的内存数，即used-buffers-cached+ buffers/cache：(可用)的内存数，即free+buffers+cachedBuffer Cache用于针对磁盘块的读写；Page Cache用于针对文件inode的读写，这些Cache能有效地缩短I/O系统调用的时间。对操作系统来说free/used是系统可用/占用的内存；对应用程序来说-/+ buffers/cache是可用/占用内存,因为buffers/cache很快就会被使用。

We should look at the application from the perspective of our work.

1.3 Hard Drive View 1, view hard disk and partition information:

fdisk -l

2. Check the disk space usage of the file system:

df -h

3. Check the I/O performance of the hard disk (show once every second, show 5 times):

iostat -x 1 5

The Iostat is included in the package Systat and can be installed with the Yum-y install Systat.

Parameters that are commonly followed:

如%util接近100%,说明产生的I/O请求太多，I/O系统已经满负荷，该磁盘可能存在瓶颈。如idle小于70%，I/O的压力就比较大了，说明读取进程中有较多的wait。

4. Check the size of a directory on your Linux system:

du -sh /root

If you find that a partition is nearly exhausted, you can go to the mount point of the partition, use the following command to find the file or directory that occupies the most space , and then find the top 10 files or directories in the system that occupy the most space in the order of large to small:

du -cksh *|sort -rn|head -n 10

1.4 Viewing Average load

Sometimes the system responds slowly, but cannot find the reason, then it is necessary to look at the average load, see if it has a large number of processes in the queue waiting.

The simplest of commands:

uptime--查看过去的1分钟、5分钟和15分钟内进程队列中的平均进程数量。

and a dynamic command top.
We only care about the following sections:

top - 21:33:09 up  1:00,  1 user,  load average: 0.00, 0.01, 0.05如果每个逻辑cpu当前的活动进程不大于3，则系统性能良好；如果每个逻辑cpu当前的活动进程不大于4，表示可以接受；如果每个逻辑cpu当前的活动进程大于5，则系统性能问题严重。

General calculation method: Load value/number of logical CPUs

You can also combine the Vmstat command to determine whether the system is busy, where:

procsr：等待运行的进程数。b：处在非中断睡眠状态的进程数。w：被交换出去的可运行的进程数。memeoryswpd：虚拟内存使用情况，单位为KB。free：空闲的内存，单位为KB。buff：被用来作为缓存的内存数，单位为KB。swapsi：从磁盘交换到内存的交换页数量，单位为KB。so：从内存交换到磁盘的交换页数量，单位为KB。iobi：发送到块设备的块数，单位为KB。bo：从块设备接受的块数，单位为KB。systemin：每秒的中断数，包括时钟中断。cs：每秒的环境切换次数。cpu按cpu的总使用百分比来显示。us：cpu使用时间。sy：cpu系统使用时间。id：闲置时间。

1.5 Other parameters

查看内核版本号：uname -a简化命令：uname -r查看系统是32位还是64位的：file /sbin/init查看发行版：cat /etc/issue或lsb_release -a查看系统已载入的相关模块：lsmod查看pci设置：lspci

Two Linux server performance evaluation 2.1.1 Factors that affect Linux server performance 1. Operating system level

CPU内存磁盘I/O带宽网络I/O带宽

2. Program application level 2.1.2 System performance evaluation standard

Impact Performance Factors	Good	Bad	Bad
Cpu	user% + sys%< 70%	user% + sys%= 85%	user% + sys% >=90%
Memory	Swap in (SI) =0 swap out (SO) =0	Per CPU with ten page/s	More swaps in & swaps out
Disk	Iowait% < 20%	Iowait% =35%	Iowait% >= 50%

which

%user：表示CPU处在用户模式下的时间百分比。%sys：表示CPU处在系统模式下的时间百分比。%iowait：表示CPU等待输入输出完成时间的百分比。swap in：即si，表示虚拟内存的页导入，即从SWAP DISK交换到RAMswap out：即so，表示虚拟内存的页导出，即从RAM交换到SWAP DISK

2.1.3 System performance analysis tools 1. Common system commands

Vmstat, SAR, Iostat, netstat, free, PS, top, etc.

2. Common Combination method

vmstat、sar、iostat检测是否是CPU瓶颈free、vmstat检测是否是内存瓶颈iostat检测是否是磁盘I/O瓶颈netstat检测是否是网络带宽瓶颈

2.1.4 Linux performance Evaluation and optimization

Overall system performance evaluation (uptime command)
uptime

16:38:00 up 118 days, 3:01, 5 users,load average:1.22, 1.02, 0.91

Attention:

• Load average three-value size is generally not larger than the number of system CPUs.

  The system has 8 CPUs, such as load average three long-term greater than 8, indicating that the CPU is busy, the load is high, may affect system performance.

• But occasionally greater than 8, generally does not affect system performance.

• If the load average output value is less than the number of CPUs, it means that the CPU has idle time slices, such as the output in this example, the CPU is very idle

2.2.1 CPU Performance Evaluation 1. Monitor the system CPU with the Vmstat command

Display the system's various resources related performance brief information, mainly look at the CPU load situation.

The following is the output of the Vmstat command on a system:

[[email protected] ~]#vmstat 2 3procs ———–memory———- —swap– —–io—- –system– —–cpu——r  b swpd freebuff  cache si so bi bo incs us sy idwa st0  0 0 162240 8304 67032 0 0 13 21 1007 23 0 1 98 0 00  0 0 162240 8304 67032 0 0 1 0 1010 20 0 1 100 0 00  0 0 162240 8304 67032 0 0 1 1 1009 18 0 1 99 0 0Procs

r--the number of processes running and waiting for CPU time slices, this value if longer than the number of system CPUs, indicating that the CPU is not enough, need to increase the CPU

b--the number of processes waiting on the resource, such as waiting for I/O, or memory exchange.

Cpuus

Percentage of CPU time consumed by the user process.
When the value of us is higher, the user process consumes more CPU time, but if the long-term is greater than 50%, the optimizer or algorithm needs to be considered.

Sy

Percentage of CPU time consumed by the kernel process. When the value of SY is high, it indicates that the kernel consumes a lot of CPU resources.

Based on experience, the reference value for Us+sy is 80%, and if Us+sy is greater than 80%, there may be insufficient CPU resources.

2. Monitor system CPU with SAR command

The SAR makes separate statistics on each aspect of the system, but increases the overhead, but the overhead can be evaluated without much impact on the system's statistical results.

The following is the CPU statistics output of a system by the SAR command:

[[email protected] ~]# sar -u 3 5Linux 2.6.9-42.ELsmp (webserver) 11/28/2008_i686_ (8 CPU)11:41:24 AM CPU %user %nice%system %iowait %steal %idle11:41:27 AM all 0.88 0.00 0.29 0.00 0.00 98.8311:41:30 AM all 0.13 0.00 0.17 0.21 0.00 99.5011:41:33 AM all 0.04 0.00 0.04 0.00 0.00 99.9211:41:36 AM all 90.08 0.00 0.13 0.16 0.00 9.6311:41:39 AM all 0.38 0.00 0.17 0.04 0.00 99.41Average: all 0.34 0.00 0.16 0.05 0.00 99.45

The output is interpreted as follows:

%user列显示了用户进程消耗的CPU 时间百分比。%nice列显示了运行正常进程所消耗的CPU 时间百分比。%system列显示了系统进程消耗的CPU时间百分比。%iowait列显示了IO等待所占用的CPU时间百分比%steal列显示了在内存相对紧张的环境下pagein强制对不同的页面进行的steal操作 。%idle列显示了CPU处在空闲状态的时间百分比。问题

Have you ever experienced a low overall CPU utilization of the system and slow application?

In a multi-CPU system, if the program uses a single thread, there will be a phenomenon, the overall CPU utilization is not high, but the system application is slow response, this may be because the program uses a single thread, the single thread uses only one CPU, resulting in a CPU utilization of 100%, unable to process other requests , while the rest of the CPU is idle, which results in a low overall CPU usage, and the application of slow phenomenon occurs.

2.3.1 Memory Performance Evaluation 1. Use the free command to monitor memory

Free is the most commonly used instruction for monitoring Linux memory usage, as shown in the following output:

[[email protected] ~]# free -mtotal used freeshared buffers cachedMem: 8111 7185 926 0 243 6299 -/+ buffers/cache: 643 7468Swap: 8189 0 8189

Empirical formula:

应用程序可用内存/系统物理内存>70%，表示系统内存资源非常充足，不影响系统性能;应用程序可用内存/系统物理内存<20%，表示系统内存资源紧缺，需要增加系统内存;20%<应用程序可用内存/系统物理内存<70%，表示系统内存资源基本能满足应用需求，暂时不影响系统性能

2. Monitor memory with the Vmstat command

[[email protected] ~]# vmstat 2 3procs ———–memory———- —swap– —–io—- –system– —–cpu——r b swpd freebuff cache si so bi bo incs us sy idwa st0 0 0 162240 8304 67032 0 0 13 21 1007 23 0 1 98 0 00 0 0 162240 8304 67032 0 0 1 0 1010 20 0 1 100 0 00 0 0 162240 8304 67032 0 0 1 1 1009 18 0 1 99 0 0

Memory

swpd--切换到内存交换区的内存数量（k为单位)。如swpd值偶尔非0，不影响系统性能free--当前空闲的物理内存数量（k为单位）buff--buffers cache的内存数量，一般对块设备的读写才需要缓冲cache--page cached的内存数量

Generally as a file system cached, frequently accessed files will be cached, such as the cache value is large, indicating that the number of cached files, if at this time the bi in IO is relatively small, the file system efficiency is better.

Swap

si--由磁盘调入内存，也就是内存进入内存交换区的数量。so--由内存调入磁盘，也就是内存交换区进入内存的数量。

The value of Si and so is not 0 for a long time, indicating insufficient system memory. system memory needs to be increased.

2.4.1 Disk I/O performance evaluation 1. Disk Storage Fundamentals

frequently accessed files or data with memory read-write instead of direct disk I/O, is highly efficient.

Separate the frequently read-write files from the long-term unchanging files and place them on different disk devices.

For data that is frequently written, consider using a bare device instead of a file system.

Bare Device Advantages:

数据可直接读写，不需经过操作系统级缓存，节省内存资源，避免内存资源争用;避免文件系统级维护开销，如文件系统需维护超级块、I-node等;避免了操作系统cache预读功能，减少了I/O请求

The disadvantages of using a bare device are:

Data management, space management is not flexible, requires a very professional people to operate.

2. Evaluating disk Performance with Iostat

[[email protected] ~]# iostat -d 2 3Linux 2.6.9-42.ELsmp (webserver) 12/01/2008_i686_ (8 CPU) Device: tps Blk_read/sBlk_wrtn/sBlk_read Blk_wrtnsda 1.87 2.58 114.12 6479462 286537372 Device: tps Blk_read/sBlk_wrtn/sBlk_read Blk_wrtnsda 0.00 0.00 0.00 0 0 Device: tps Blk_read/sBlk_wrtn/sBlk_read Blk_wrtnsda 1.00 0.00 12.00 0 24

The explanations are as follows:

Blk_read/s--每秒读取数据块数Blk_wrtn/s--每秒写入数据块数Blk_read--读取的所有块数Blk_wrtn--写入的所有块数

A basic understanding of disk read and write performance is available through the blk_read/s and BLK_WRTN/S values.
If the BLK_WRTN/S value is large, indicating that disk writes are frequent, consider optimizing the disk or program,
If the blk_read/s value is large, it means that there are many direct disk read operations, which can put the read data into memory

Rules follow:

Long-term, ultra-large data read and write, is certainly not normal, this situation will certainly affect the performance of the system.

3. Using SAR to evaluate disk performance

With the "sar–d" combination, you can make a basic statistic of the system's disk IO, see one of the following outputs:

[[email protected] ~]# sar -d 2 3Linux 2.6.9-42.ELsmp (webserver) 11/30/2008_i686_ (8 CPU)11:09:33 PM DEV tps rd_sec/swr_sec/savgrq-sz avgqu-sz await svctm %util11:09:35 PM dev8-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0011:09:35 PM DEV tps rd_sec/swr_sec/savgrq-sz avgqu-sz await svctm %util11:09:37 PM dev8-0 1.00 0.00 12.00 12.00 0.00 0.00 0.00 0.0011:09:37 PM DEV tps rd_sec/swr_sec/savgrq-sz avgqu-sz await svctm %util11:09:39 PM dev8-0 1.99 0.00 47.76 24.00 0.00 0.50 0.25 0.05Average: DEV tps rd_sec/swr_sec/savgrq-sz avgqu-sz await svctm %utilAverage: dev8-0 1.00 0.00 19.97 20.00 0.00 0.33 0.17 0.02

Parameter meaning:

await--平均每次设备I/O操作等待时间（毫秒）svctm--平均每次设备I/O操作的服务时间（毫秒）%util--一秒中有百分之几的时间用于I/O操作

Evaluation criteria for disk IO performance:

The normal SVCTM should be less than the await value, and SVCTM and disk performance, CPU, memory load will also affect the SVCTM value, too many requests will indirectly lead to the increase of SVCTM value.

await值取决svctm和I/O队列长度以及I/O请求模式，如果svctm的值与await很接近，表示几乎没有I/O等待，磁盘性能很好，如果await的值远高于svctm的值，则表示I/O队列等待太长，系统上运行的应用程序将变慢，此时可以通过更换更快的硬盘来解决问题。

%util--measures disk I/O important metrics,

If the%util is close to 100%, indicating that the disk generates too many I/O requests and that the I/O system is fully operational, the disk may have bottlenecks.

The program can be optimized or replaced by a higher, faster disk.

2.5.1. Network performance evaluation

（1）通过ping命令检测网络的连通性（2）通过netstat –i组合检测网络接口状况（3）通过netstat –r组合检测系统的路由表信息（4）通过sar –n组合显示系统的网络运行状态

Three Linux server performance Tuning 1. Tuning the Linux kernel elevator algorithm for disk I/O

When the file system is selected, the algorithm balances low latency requirements, collects enough data, and effectively organizes requests for disk reads and writes.

2. Disable unnecessary daemons, save memory and CPU resources

许多守护进程或服务通常非必需，消耗宝贵内存和CPU时间。将服务器置于险地。禁用可加快启动时间，释放内存。减少CPU要处理的进程数

Some Linux daemons that should be disabled are automatically run by default:

Serial Number Daemon Description
1 APMD Advanced Power Management Daemon
2 Nfslock for NFS file locking
3 ISDN ISDN Moderm support
4 Autofs automatically mount the file system in the background (e.g. Mount CD-ROM automatically)
5 Sendmail Mail Transfer agent
6 Xfs X Window font server

3. Turn off GUI4, clean up unnecessary modules or functions

Too many features or modules that are launched in a server package are actually unwanted (like many feature modules in Apache), disabling helps increase the amount of system memory available, freeing up resources for the software that really needs it to run faster.

5. Disable the Control Panel

In Linux, there are many popular control panels, such as cpanel,plesk,webmin and phpMyAdmin, which disable the release of approximately 120MB of memory and memory usage by approximately 30-40%.

6. Improve Linux Exim server performance

With the DNS cache daemon, you can reduce the bandwidth and CPU time required to resolve DNS records, and the DNS cache improves network performance by eliminating the need to find DNS records every time from the root node.

Djbdns is a very powerful DNS server with DNS caching capabilities, DJBDNS is more secure than the BIND DNS server, performs better, can be downloaded directly through http://cr.yp.to/, or is available through Red Hat packages.

7, using AES256 enhanced GPG file encryption security

To improve the security of backup files or sensitive information, many Linux system administrators use GPG encryption, in the use of GPG, it is best to specify GPG using AES256 encryption algorithm, AES256 use 256-bit key, it is an open encryption algorithm, the United States National Security Agency (NSA) Use it to protect top secret information.

8. Remote Backup service Security

Security is the most important factor in choosing a remote backup service, and most system administrators are afraid of two things: (hackers) can delete backup files and not restore the system from backup.

To ensure the security of backup files 100%, backup service companies provide remote backup servers, use SCP scripts or rsync to transfer data over SSH so that no one can access and access remote systems directly, so no one can delete data from the backup service. When choosing a remote backup service provider, it's a good idea to understand its service robustness in many ways, and if you can, you can test it yourself.

9. Update the default kernel parameter settings

To successfully and successfully run an enterprise application, such as a database server, you may need to update some of the default kernel parameter settings, for example, 2.4.x series kernel Message Queuing parameter Msgmni has a default value (for example, shared memory, or Shmmax in red The hat system has a default of only 33554432 bytes, it only allows limited database concurrent connections, and below provides some recommended values for better database server operation (from the IBM DB2 Support site):

kernel.shmmax=268435456 (32-bit)
kernel.shmmax=1073741824 (64-bit)
kernel.msgmni=1024
fs.file-max=8192
Kernel.sem= "250 32000 32 1024″

10. Optimize TCP

Optimizing the TCP protocol helps improve network throughput, the larger the bandwidth used across the WAN, and the longer the latency, the more TCP Linux size is recommended to increase the data transfer rate, and the TCP Linux size determines whether the sending host will not receive a data transfer acknowledgement How much data can be sent to the receiving host.

11. Select the correct file system

Replace ext3 with the Ext4 file system

EXT4 is an enhanced version of the Ext3 file system, extending storage limits

Log capability to ensure high levels of data integrity (in abnormal shutdown events)

It does not need to check the disk (this is a very time-consuming action) when it is not properly shut down and restarted

Faster write speed, EXT4 log optimizes drive head motion

12. Using the Noatime file system Mount option

Using the Noatime option in the File system boot profile fstab, this mount option can effectively improve performance if external storage is used.

13. Adjust Linux file descriptor limits

Linux limits the number of file descriptors that any process can open, with the default limit of 1024 per process, which may hinder benchmarking clients (such as Httperf and Apachebench) and the Web server itself from getting the best performance, with one process per connection for Apache, So it is not affected, but a single-process Web server, such as Zeus, uses one file descriptor per connection and is therefore susceptible to the default restrictions.

Opening a file limit is a restriction that can be adjusted with the Ulimit command, the ulimit-as command displays the current limit, and the Ulimit-ah command displays a hard limit (you cannot increase the limit until the kernel parameters in/proc are not adjusted).

Linux third-party application performance Tips

As with third-party applications running on Linux, there are many performance tuning techniques that can help you improve the performance of your Linux servers and lower your operating costs.

14. Correct configuration of MySQL

To allocate more memory to MySQL, you can set the MySQL cache size, and if the MySQL server instance uses more memory, reduce the cache size and increase the MySQL cache if MySQL stalls when the request grows.

15. Proper configuration of Apache

Checking how much memory Apache uses, and then adjusting the startservers and minspareservers parameters to free up more memory, will help you save 30-40% of memory.

16. Analyzing Linux Server performance

The best way to improve system efficiency is to find the bottleneck that causes the overall speed drop and solve it, here are some basic tips for finding the key bottlenecks in your system:

When a large application, such as OpenOffice and Firefox, runs simultaneously, the computer may start to slow down and the chances of running out of memory are higher.

If it is really slow to start, it may be that the initial startup of the application takes a long time to load, and it is normal to run once it is started, or the hard drive is probably too slow.

CPU load continues to be high, memory is sufficient, but CPU utilization is low, and the load time can be monitored using the CPU load analysis tool.

17. Learn 5 Linux performance commands

You can manage the performance of your Linux system with a few commands, listed below are the 5 most common Linux performance commands, including
top, Vmstat, Iostat, free, and SAR, which help system administrators quickly resolve performance issues.

(1) Top

The tasks of the current kernel service also display statistics for many host states, which are automatically updated every 5 seconds by default.
such as: current uptime, system load, number of processes and memory utilization,

In addition, this command shows the processes that use the most CPU time (including the various information for each process, such as running users, executing commands, and so on).

(2) Vmstat

The Vmstat command provides a snapshot of the current CPU, IO, process, and memory utilization, which, like the top command, automatically updates data such as:

$ vmstat 10

(3) Iostat

Iostat provides three reports: CPU utilization, device utilization, and network file system utilization, and three reports can be displayed independently using the-c,-d and-H parameters.

(4) Free

Displays the main memory and swap space memory statistics, specifying the-t parameter to display total memory, specifying the-b parameter in bytes, using-m in megabytes, and by default in kilobytes.

The free command can also be run continuously using the-s parameter plus a delay time (in seconds), such as:

$ free-s 5

(5) SAR

Collect, view, and record performance data, a command that is older than the previous commands, and collects and displays longer periods of data.

Other

Here are some of the other performance techniques that are categorized as:

18. Transfer log files to memory

When a machine is running, it is best to put the system log in memory, when the system shuts down and then copy it to the hard disk, when you run a syslog-enabled laptop or mobile device, Ramlog can help you improve the system battery or mobile device flash drive life, One of the benefits of using Ramlog is that there is no need to worry about a daemon sending a message to the syslog every 30 seconds, before the hard drive must be kept running, which is bad for hard drives and batteries.

19. Package first, then write

In memory to partition a fixed size of the space to save the log file, which means that the laptop hard drive does not have to keep running, only when a daemon needs to write to the log to run, note that the Ramlog use of memory space is fixed, otherwise the system memory will be used up quickly, if the notebook uses solid-state drive, 50-80MB memory can be allocated to ramlog use, Ramlog can reduce many write cycles, greatly improve the life of solid-state drives.

20, general tuning skills

Use static content instead of dynamic content whenever possible, and if you are generating weather forecasts or other data that must be updated every 1 hours, it is best to write a program that generates a static file every 1 hours instead of letting the user run a CGI to generate the report dynamically.

Choosing the fastest and most suitable api,cgi for a dynamic application can be the easiest to program, but it will produce a process for each request, which is usually a costly and unnecessary process, fastcgi is a better choice, like Apache Mod_perl, Can greatly improve the performance of your application.

Linux server performance Analysis and tuning