Troubleshooting of insufficient system memory caused by excessive memory usage by Linux Server Cache
Problem description
When the memory usage of the Linux Server exceeds the threshold, an alarm is triggered.
Troubleshooting
First, use the free command to observe the system memory usage, as shown below:
total used free shared buffers cachedMem: 24675796 24587144 88652 0 357012 1612488-/+ buffers/cache: 22617644 2058152Swap: 2096472 108224 1988248
It can be seen that the total memory size is 24675796KB, 22617644KB is used, and only 2058152KB is left.
Then, run the top Command, shift + M sort by memory, observe the process with the maximum memory usage in the system, and find that only 18 GB memory is occupied, and other processes are very small, can be ignored.
Therefore, where can I use nearly 4 GB of memory (about 4 GB?
Further, we found through cat/proc/meminfo that there was nearly 4 GB (3688732 KB) of Slab Memory:
......Mapped: 25212 kBSlab: 3688732 kBPageTables: 43524 kB......
Slab is used to store the kernel data structure cache, and then use the slabtop command to view the memory usage:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 13926348 13926348 100% 0.21K 773686 18 3494744K dentry_cache334040 262056 78% 0.09K 8351 40 33404K buffer_head151040 150537 99% 0.74K 30208 5 120832K ext3_inode_cache
It is found that most of the data (about 3.5 GB) is used for dentry_cache.
Problem Solving
1. Modify/proc/sys/vm/drop_caches to release the cache memory space occupied by Slab (refer to the official drop_caches documentation ):
Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.To free pagecache:* echo 1 > /proc/sys/vm/drop_cachesTo free dentries and inodes:* echo 2 > /proc/sys/vm/drop_cachesTo free pagecache, dentries and inodes:* echo 3 > /proc/sys/vm/drop_cachesAs this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed.This tunable was added in 2.6.16.
2. method 1 requires the user to have the root permission. If the user is not the root but has the sudo permission, you can use the sysctl command to set it:
$sync$sudo sysctl -w vm.drop_caches=3$sudo sysctl -w vm.drop_caches=0 #recovery drop_caches
After the operation, you can use sudo sysctl-a | grep drop_caches to check whether the operation takes effect.
3. Modify/proc/sys/vm/vfs_cache_pressure and adjust the priority (100 by default) for clearing inode/dentry caches. There are related explanations in LinuxInsight:
At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.
For specific setting methods, refer to method 1 or method 2.
References
Https://www.kernel.org/doc/Documentation/sysctl/vm.txt
Http://major.io/2008/12/03/reducing-inode-and-dentry-caches-to-keep-oom-killer-at-bay/
Http://linux-mm.org/Drop_Caches
For more details, please continue to read the highlights on the next page: