Troubleshooting of Linux system memory insufficiency caused by excessive Cache Usage

Problem description

When the memory usage of the Linux Server exceeds the threshold, an alarm is triggered.

Troubleshooting

First, use the free command to observe the system memory usage, as shown below:

 
 

  
  
total       used       free     shared    buffers     cached 
  
  
Mem:      24675796   24587144      88652          0     357012    1612488 
  
  
-/+ buffers/cache:   22617644    2058152 
  
  
Swap:      2096472     108224    1988248 
 
 

It can be seen that the total memory size is 24675796KB, 22617644KB is used, and only 2058152KB is left.

Then, run the top Command, shift + M sort by memory, observe the process with the maximum memory usage in the system, and find that only 18 GB memory is occupied, and other processes are very small, can be ignored.

Therefore, there is nearly 4 GB of memory 22617644KB-18GB, about 4 GB) Where is it used?

Further, we found through cat/proc/meminfo that there was nearly 4 gb213732 KB) Slab Memory:

 
 

  
  
...... 
  
  
Mapped:          25212 kB 
  
  
Slab:          3688732 kB 
  
  
PageTables:      43524 kB 
  
  
...... 
 
 

Slab is used to store the kernel data structure cache, and then use the slabtop command to view the memory usage:

 
 

  
  
OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME 
  
  
13926348 13926348 100%    0.21K 773686       18   3494744K dentry_cache 
  
  
334040 262056  78%    0.09K   8351       40     33404K buffer_head 
  
  
151040 150537  99%    0.74K  30208        5    120832K ext3_inode_cache 
 
 

It is found that most of them are about 3.5 GB) and are used for dentry_cache.

Problem Solving

1. Modify/proc/sys/vm/drop_caches to release the cache memory occupied by Slab. For details, refer to the official drop_caches documentation ):

 
 

  
  
Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free. 
  
  
To free pagecache: 
  
  
* echo 1 > /proc/sys/vm/drop_caches 
  
  
To free dentries and inodes: 
  
  
* echo 2 > /proc/sys/vm/drop_caches 
  
  
To free pagecache, dentries and inodes: 
  
  
* echo 3 > /proc/sys/vm/drop_caches 
  
  
As this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed. 
  
  
This tunable was added in 2.6.16. 
 
 

2. method 1 requires the user to have the root permission. If the user is not the root but has the sudo permission, you can use the sysctl command to set it:

 
 

  
  
$sync 
  
  
$sudo sysctl -w vm.drop_caches=3 
  
  
$sudo sysctl -w vm.drop_caches=0 #recovery drop_caches 
 
 

After the operation, you can use sudo sysctl-a | grep drop_caches to check whether the operation takes effect.

3. Modify/proc/sys/vm/vfs_cache_pressure and adjust the priority of inode/dentry caches to 100 by default. There are related explanations in LinuxInsight:

 
 

  
  
At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a “fair” rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.  
 
 

For specific setting methods, refer to method 1 or method 2.

References

• Https://www.kernel.org/doc/Documentation/sysctl/vm.txt
• Http://major.io/2008/12/03/reducing-inode-and-dentry-caches-to-keep-oom-killer-at-bay/
• Http://linux-mm.org/Drop_Caches

The following records the progress of further troubleshooting.

Deeper reasons

I have found that there is a large amount of dentry_cache memory in Linux. Why is there so much dentry_cache?

1. first, clarify the concept and function of dentry_cache: Directory item high-speed cache is designed by Linux to improve the processing efficiency of directory item objects. It records the ing relationship between directory items and inode. Therefore, when an application initiates a stat system call, it will create the corresponding dentry_cache item. If each stat file does not exist, there will always be a large number of new dentry_cache items created ).

2. the current server is a node of the storm cluster. First, I thought of storm-related working processes. strace the storm worker Process found that there were very frequent stat System calls, in addition, the stat file is always a new file name:

sudo strace -fp <pid> -e trace=stat

3. It is further observed that storm's worker process will frequently create, open, close, and delete heartbeat files in the local directory, with a new file name every second:

sudo strace -fp <pid> -e trace=open,stat,close,unlink

The above is why there are so many dentry_caches in the system.

A strange phenomenon

By observing/proc/meminfo, we found that slab Memory is divided into two parts:

SReclaimable // recoverable slabSUnreclaim // unrecoverable slab

At that time, the status of the server was: Some of the memory occupied by slab, most of which were displayed as SReclaimable, that is, they could be recycled.

However, we can see through slabtop that the most important part of the slab memory is dentry_cache. OBJS is almost ACTIVE, and 100% is in use.

OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   13926348 13926348 100%    0.21K 773686       18   3494744K dentry_cache334040 262056  78%    0.09K   8351       40     33404K buffer_head151040 150537  99%    0.74K  30208        5    120832K ext3_inode_cache

Why is it recycled but ACTIVE? Ask the Linux kernel experts to give an enthusiastic explanation:

Will dcache not be automatically recycled and released due to its ACTIVE status?

Allow the system to automatically recycle dcache

As we mentioned in the previous section, most of the slab memory on the server is SReclaimable. Can we give the OS a chance to automatically trigger the recycle operation at a certain time? The answer is yes.

I checked some information about Linux dcache and found that the operating system will trigger the kswapd kernel process to release it after the memory critical threshold is reached. The calculation method of this threshold is as follows:

1. First, grep low/proc/zoneinfo. the following result is obtained:

        low      1        low      380        low      12067

2. this threshold is used to add up the above three columns and multiply them by 4 kb. After calculation, it is found that the recovery threshold of the current server is only 48 MB, so it is difficult to see this phenomenon, in reality, the operating system will fail to respond to hang if it does not wait for recovery.

3. you can use the following method to increase the threshold: vm. extra_free_kbytes is set to vm. min_free_kbytes is as large as it is, the low threshold in/proc/zoneinfo is doubled, and the high threshold is also increased, and so on.

$ sudo sysctl -a | grep free_kbytes       vm.min_free_kbytes = 39847vm.extra_free_kbytes = 0
$ sudo sysctl -w vm.extra_free_kbytes=836787  ######1GB

4. for example, when the low threshold is set to 1 GB, when the free memory of the system is smaller than 1 GB, it is observed that the kswapd process starts to work from Sleeping to Running ), at the same time, dcache starts to be recycled by the system until the free memory of the system is between the low threshold and the high threshold.

Http://www.cnblogs.com/panfeng412/p/drop-caches-under-linux-system-2.html