php-cgi too many processes, resulting in system resource exhaustion

1:top command to see how the system's CPU RAM swap is used

By analysis, you can see a total of 602 processes, of which 601 processes hibernate. This seems to be something wrong, the kernel process is about 80, plus memcached, Nginx, mysqld, will not exceed 90. In addition to these, only php-fpm management of the php-cgi, is it ...?

CPU display, CPU pressure is not big, can say no pressure. We looked at the memory usage profile again, found that 4G of memory, the consumption of the remaining few (free+buffers), more than 95% of the memory has been allocated. The use of interactive space, we temporarily do not care. The command top also lists the most resource-intensive processes, and the longest-running (time+) mysqld (about 2 hours) consumes less resources. In addition, looking at php-cgi, a single php-cgi occupies a lot of memory. So, you can be bold to guess: server memory resources are relatively tense, and not by a process to occupy a large amount of memory, it is possible that some suspended processes occupy memory is not released. Verify our ideas by further monitoring memory usage with free.

2: View RAM usage

Let's take a look at the MEM statistics, total represents the amount of physical memory, about 4G. Used, which indicates that memory has been allocated, is not represented as being used, including (buffer&cached). Free refers to unallocated memory, and buffers and cached represent memory that has been allocated but not yet in use. The second line (Buffers/cache), used means that the memory is actually used, is obtained by the first line (used-buffer-cached), and free indicates that memory that has not been used is obtained by the first line (free+buffer+cached). Swap lines represent memory exchange usage, with little (infrequently) swpd, which does not affect server performance because the system needs to swap the V-type memory pages out or adjust the size of buffer and cached. However, frequent swpd may mean that the server is running out of physical memory, less than the specified swap rating, and needs to be swapped out of memory pages.

When we view the free results, we mainly look at the second line. A glance can see 4G of memory, of which 3898M memory is used, and 49M of memory is not, are running out. This also confirms our first step of guessing that memory is exhausted. Here, we further suspect that the memory space is severely insufficient, the process will be blocked, the system will continue to exchange the unused data out so, the data will be used to read into the SI. We can further verify through the vmstat that this conjecture of ours.

3:vmstat monitoring the use of memory

As a memory monitor, we are more concerned about SWPD, free, si, and so. When the general system is not busy, we see that the value of SWPD,SO will not last very high, often 0. Here, we see that the SWPD value is 1.5G, and the free value is small, again indicating that there is not enough physical memory. Where SI reports the total amount of memory moved from swap to physical memory per second, so reports the total amount of memory moved from physical memory to swap area per second. Of course, si is sometimes large, and do not excessive anxiety, often encountered a program requires large memory to read and write media files, Si value will become larger. So, it is usually a memory shortage of a signal, if the long time this value has remained large, it is likely that the memory is not enough, small fluctuations can be ignored. Next, you can find the culprit that consumes memory through PS.

4:ps find the culprit that consumes memory

Instruction PS is more commonly used, but also relatively simple. With the results reported above, we can hit php-cgi this process at a glance. Although a single php-cgi memory is not too large, but the 503 php-cgi process, it is a bit scary. It takes up almost all of the memory (503*0.3%). We can guess that php-cgi is managed by PHP-FPM and is not properly configured to php-fpm one of the parameters, resulting in an excessive number of php-cgi processes being opened.

php-cgi too many processes, resulting in system resource exhaustion