Troubleshooting of insufficient system memory caused by excessive memory usage by Linux Server Cache

Troubleshooting of insufficient system memory caused by excessive memory usage by Linux Server Cache
Problem description

When the memory usage of the Linux Server exceeds the threshold, an alarm is triggered.

Troubleshooting

First, use the free command to observe the system memory usage, as shown below:

             total       used       free     shared    buffers     cachedMem:      24675796   24587144      88652          0     357012    1612488-/+ buffers/cache:   22617644    2058152Swap:      2096472     108224    1988248

It can be seen that the total memory size is 24675796KB, 22617644KB is used, and only 2058152KB is left.

Then, run the top Command, shift + M sort by memory, observe the process with the maximum memory usage in the system, and find that only 18 GB memory is occupied, and other processes are very small, can be ignored.

Therefore, where can I use nearly 4 GB of memory (about 4 GB?

Further, we found through cat/proc/meminfo that there was nearly 4 GB (3688732 KB) of Slab Memory:

......Mapped:          25212 kBSlab:          3688732 kBPageTables:      43524 kB......

Slab is used to store the kernel data structure cache, and then use the slabtop command to view the memory usage:

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   13926348 13926348 100%    0.21K 773686       18   3494744K dentry_cache334040 262056  78%    0.09K   8351       40     33404K buffer_head151040 150537  99%    0.74K  30208        5    120832K ext3_inode_cache

It is found that most of the data (about 3.5 GB) is used for dentry_cache.

Problem Solving

1. Modify/proc/sys/vm/drop_caches to release the cache memory space occupied by Slab (refer to the official drop_caches documentation ):

Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.To free pagecache:* echo 1 > /proc/sys/vm/drop_cachesTo free dentries and inodes:* echo 2 > /proc/sys/vm/drop_cachesTo free pagecache, dentries and inodes:* echo 3 > /proc/sys/vm/drop_cachesAs this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed.This tunable was added in 2.6.16.

2. method 1 requires the user to have the root permission. If the user is not the root but has the sudo permission, you can use the sysctl command to set it:

$sync$sudo sysctl -w vm.drop_caches=3$sudo sysctl -w vm.drop_caches=0 #recovery drop_caches

After the operation, you can use sudo sysctl-a | grep drop_caches to check whether the operation takes effect.

3. Modify/proc/sys/vm/vfs_cache_pressure and adjust the priority (100 by default) for clearing inode/dentry caches. There are related explanations in LinuxInsight:

At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

For specific setting methods, refer to method 1 or method 2.

References

Https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Http://major.io/2008/12/03/reducing-inode-and-dentry-caches-to-keep-oom-killer-at-bay/

Http://linux-mm.org/Drop_Caches

For more details, please continue to read the highlights on the next page:

1 2 Next Page