Alarm triggered by a Laravel queue

Alarm triggered by a Laravel queue

An alarm is reported on one server. The memory usage is too high. It is strange that other servers in the cluster are okay. However, from past experiences, there is a strange answer behind every incredible question.

First, check the memory usage through "free-m" and find that 6893 M is used, with 976 M left:

Free

Then, click "top" to check which processes occupy a large amount of memory and sort by memory using "shift + m:

Top

Although we can confirm that the available memory of the system is insufficient through the free command, but through the top command we did not find any process with large memory usage, where is the memory?

We mentioned at the beginning that there is a problem with only one server in the cluster and other servers are normal, so we compared the process list of the problematic server and the normal server, several processes are added to the problematic Server:

/usr/local/bin/php artisan queue:listen
/usr/local/bin/php artisan queue:work

After confirmation, they are Laravel queues. Although intuition tells me that the problem is associated with it, the process itself does not occupy much memory, and the cause cannot be confirmed immediately, we use the exclusion method to switch the queue to another normal server to see if the problem will be reproduced. After a while, the same problem occurs again.

Since commands such as free and top cannot determine the memory destination, let's look at "meminfo 」:

Meminfo

As shown in, a large amount of memory is consumed by Slab. Further, it is consumed by SReclaimable. That is to say, the memory is consumed by some reusable Slab. For details, refer to "slabtop:

Slabtop

Dentry has basically been used up. If you are the same as me, you don't know what it means. Search, use Google to flip the wall, and use AOL instead of flip the wall. For details, refer to the following:

1. Troubleshooting of insufficient system memory caused by excessive memory usage by Linux Server Cache

In short, the memory dentry caches the information of recently accessed files. If a large number of files are frequently operated, dentry will continue to increase, so the problem becomes to check whether the Laravel queue has similar problems.

As mentioned above, the Laravel queue has a listen process and a work process. We can determine from the name that the former is the main process and the latter is a sub-process, the sub-process is a working process. However, when I directly trace the sub-process through strace, I am prompted that the sub-process does not exist. Further debugging found that the sub-process will be restarted constantly!

Since it is difficult for us to directly track sub-processes, we can start with the parent process to track the file operations of sub-processes:

shell> strace -f -e trace=open,close,stat,unlink -p $(    ps aux | grep "[q]ueue:listen" | awk '{print $2}')

It is a pity that Laravel itself is known as a master framework and relies on a lump of files. Therefore, the tracking results are filled with a large number of normal open, close, and stat operations of framework files, try to track only unlik operations:

shell> strace -f -e trace=unlink -p $(    ps aux | grep "[q]ueue:listen" | awk '{print $2}')

The Laravel queue is frequently used to delete files. Each time the sub-process is restarted, the deletion is performed:

unlink(“/tmp/.ZendSem.aXaa3Z”)
unlink(“/tmp/.ZendSem.teQG0Y”)
unlink(“/tmp/.ZendSem.Bn3ien”)
unlink(“/tmp/.ZendSem.V4s8RX”)
unlink(“/tmp/.ZendSem.PnNuTN”)

Therefore, a large amount of dentry cache is consumed. Read the Laravel queue documentation and find that the Laravel queue actually provides a process mode that does not restart. In this way, a large number of temporary files are not created frequently, and a large number of dentry caches are not consumed. We recommend that you use this mode.

If it is unavoidable to frequently create a large number of temporary files, we can delete recoverable slab (including dentries and inodes) by setting drop_caches to 2 as described in the Linux document ), relatively crude:

shell> echo 2 > /proc/sys/vm/drop_caches

In addition, you can set vfs_cache_pressure to be greater than 100 to increase the recovery probability, which is more gentle:

shell> echo 1000 > /proc/sys/vm/vfs_cache_pressure

According to the test results, the vfs_cache_pressure function is limited. Of course, it may be because my posture is incorrect. Since you are gentle and disobedient, don't blame me! Min_free_kbytes is also mentioned in some materials. However, we recommend that you be careful with this parameter because it is dangerous. If you are interested, you can check the relevant information on your own.

This article permanently updates the link address: