Analysis on system resource occupation by PHP-FPM

The tester told us that games on the Facebook platform are obviously slower than those on other platforms. I asked if it was caused by a wall network. They said: No, other games are also smooth. Using httpwatch to view HTTP requests, we found that API. php requests took 6 seconds and most of the time was red (httpwatch red indicates the server operation time ). Usually 2 seconds. Now 6 seconds, there are so many differences in the server processing time on different servers. Intuition tells: Server resources.

Server Overview

• Operating System: centos5.3-64-2.6.18-164.el5
• Application Services nginx0.6, PHP-FPM, mysql5.1, php5.2, and memcached
• Memcached only configures cached game system settings and template parameters.

Fault Detection

1. No matter what else, first top the CPU, Ram, swap which is relatively tight.

From the analysis, we can see that there are 602 processes, of which 601 are sleep. It seems that something is wrong, and there are about 80 kernel processes, and there will be no more than 90 memcached, nginx, and mysqld. Aside from this, only PHP-CGI managed by PHP-FPM is left. Is it ...?
CPU shows that the CPU pressure is not high, so there is no pressure. Let's look at the memory usage summary. We found that 4 GB memory consumed less (free + buffers), and more than 95% of the memory has been allocated. We do not care about the use of interactive spaces for the moment. Command top also lists the processes that consume the most resources. The mysqld (about 2 hours) that run for the longest time (time +) does not occupy the most resources. In addition, let's look at PhP-CGI again. The memory occupied by a single PHP-CGI is not much. Therefore, we can boldly guess that the memory resources on the server are relatively tight, and a process does not occupy a large amount of memory. Some suspended processes may occupy the memory and are not released. Use free to further monitor memory usage and verify our ideas.
2. Run the free command to check the usage of Ram resources. Of course, you can also view the file/proc/meminfo

Let's take a look at the mem statistics. Total indicates the total physical memory, which is about 4 GB. Used indicates that the memory has been allocated. If it is allocated, it does not indicate that it is used, including (buffer & cached ). Free refers to unallocated memory. buffers and cached indicate allocated but not used memory. In the second row (buffers/cache), used indicates that the memory is actually used, which is obtained from the first row (used-buffer-cached, free indicates the memory that has not been used, which is obtained from the first row (free + buffer + cached. The swap row indicates the memory swap usage. A small number of (less frequent) SWPD will not affect the server performance, because the system needs to swap out the V-type memory page or adjust the buffer and cached size. However, frequent SWPD may mean that the physical memory of the server is insufficient, which is smaller than the specified swap rating and needs to be swapped out of the Memory Page.

When viewing the free result, we mainly look at the second line. At a glance, we can see 4 GB memory, of which M memory is used, and 49 M memory is not used up. This also confirms our first conjecture that the memory is used up. Here, we further guess that when the memory space is seriously insufficient, the process will be blocked, and the system will constantly replace unused data with so and read the data to SI. We can use vmstat to further verify our conjecture.

3. Command vmstat to monitor memory usage

As for memory monitoring, we are more concerned with SWPD, free, Si, and so. Generally, when the system is not busy, we can see that the value of SWPD and so does not continue to be very high and is often 0. Here, we can see that the value of SWPD is 1.5 GB, and the value of free is small, again indicating that the physical memory is insufficient. Among them, Si reports the total amount of memory transferred from the swap area to the physical memory per second, and so reports the total amount of memory transferred from the physical memory to the swap area per second. Of course, Si is sometimes too big and don't worry too much. When a program needs a large memory to read and write media files, the SI value will increase. On the other hand, it is usually a signal of memory shortage. If this value remains large for a long time, it is likely that the memory is insufficient and the amount of data fluctuates slightly, so you can ignore it. Next, we can find out the culprit of memory consumption through PS.
4. Command PS to find out the culprit of memory consumption

[Root @ localhost ~] # Ps-a-sort-RSS-O comm, pmem, pcpu | uniq-c | head-15
1 command % mem % CPU
1 mysqld 0.6 0.0
503 PHP-CGI 0.3 0.0
5 PHP-CGI 0.2 0.0
1 PHP-CGI 0.1 0.0
1 PHP-CGI 0.0 0.0
1 memcached 0.0 0.0
1 sshd 0.0 0.0
1 nginx 0.0 0.0
1 sshd 0.0 0.0
1 nginx 0.0 0.0
2 bash 0.0 0.0
3 nginx 0.0 0.0
1 sshd 0.0 0.0
1 nginx 0.0 0.0 command PS is relatively common and simple. The above report results allow us to hit the PHP-CGI process at a glance. Although the memory occupied by a single PHP-CGI is not too large, the 503 PHP-CGI processes are a little scary. Almost all the memory is occupied (503 * 0.3% ). We can guess that PHP-CGI is managed by PHP-FPM. Is it possible to enable too many PHP-CGI processes due to improper configuration of a certain parameter of PHP-FPM.
5. Set the PHP-FPM process count management

By setting the max_children value of the php-conf.conf to 150 again, the system memory is restored to normal use. Free, Si, so, and B indicate that the memory system resources are normal and there is no pressure.

The memory released by the PHP-CGI process is not immediately recycled by the system. a php-CGI occupies about 20 mb of memory (depending on the PHP extensions you load ). Therefore, it is necessary to limit the number of PHP-CGI processes you have started. In this case, you can use top to calculate the number of PHP-CGI during server peak hours. You can also use netstat-NP | grep 127.0.0.1: 9000 to collect data as suggested by PHP-FPM. Set max_children to minimize the number of waits.

6. Memory occupied by a PHP-CGI
A PHP-CGI process occupies about 20 mb of memory (depending on how many modules Your PHP has loaded ). You can use the pmap command to check the memory occupied. Therefore, try not to load unnecessary PHP extension modules to reduce unnecessary memory waste.

[Root @ localhost etc] # pmap $ (pgrep PHP-CGI | head-1)
6746:/usr/local/PHP/bin/PHP-CGI-FPM-config/usr/local/PHP/etc/php-fpm.conf
0000000000400000 6680 k r-X-/usr/local/PHP/bin/PHP-CGI
20171000000c86000 268 k rw-/usr/local/PHP/bin/PHP-CGI
20171000000cc9000 56 k rw-[anon]
0000000005012000 2240 k rw-[anon]
2017003efd200000 112 k r-X-/lib64/ld-2.5.so
.......
20172ac28a7a5000 2048 k --/usr/local/PHP/lib/PHP/extensions/no-debug-non-zts-20060613/xhprof. So
4102ac28a9a5000 4 k rw-/usr/local/PHP/lib/PHP/extensions/no-debug-non-zts-20060613/xhprof. So
20172ac28a9a6000 84 k r-X-/usr/local/PHP/lib/PHP/extensions/no-debug-non-zts-20060613/APC. So
Export 2ac28a9bb000 2048 k --/usr/local/PHP/lib/PHP/extensions/no-debug-non-zts-20060613/APC. So
4102ac28abbb000 8 k rw-/usr/local/PHP/lib/PHP/extensions/no-debug-non-zts-20060613/APC. So
20172ac28abbd000 32 k rw-[anon]
20172ac28abd4000 40 k r-X-/lib64/libnss_files-2.5.so
20172ac28abde000 2044 k --/lib64/libnss_files-2.5.so
20172ac28addd000 4 k r --/lib64/libnss_files-2.5.so
4102ac28adde000 4 k rw-/lib64/libnss_files-2.5.so
20177fffa717e000 84 k RW-[Stack]
Ffffffffff600000 8192 k -- [anon]
Total 154172 K