Problem Description: The Linux server's memory usage exceeds the threshold and an alarm is triggered. First, use the free command to observe the memory usage of the system, which is displayed as follows: totalusedfreesharedbufferscachedMem: 24675796245871448865203570121612488-/+ Linux system
Problem Description
When the memory usage of the Linux server exceeds the threshold, an alarm is triggered.
Troubleshooting
First, use the free command to observe the system memory usage, as shown below:
- total used free shared buffers cached
- Mem: 24675796 24587144 88652 0 357012 1612488
- -/+ buffers/cache: 22617644 2058152
- Swap: 2096472 108224 1988248
It can be seen that the total memory size is 24675796KB, 22617644KB is used, and only 2058152KB is left.
Then, run the top command, shift + M sort by memory, observe the process with the maximum memory usage in the system, and find that only 18 GB memory is occupied, and other processes are very small, can be ignored.
Therefore, where can I use nearly 4 GB of memory (about 4 GB?
Further, we found through cat/proc/meminfo that there was nearly 4 GB (3688732 KB) of Slab memory:
- ......
- Mapped: 25212 kB
- Slab: 3688732 kB
- PageTables: 43524 kB
- ......
Slab is used to store the kernel data structure cache, and then use the slabtop command to view the memory usage:
- OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
- 13926348 13926348 100% 0.21K 773686 18 3494744K dentry_cache
- 334040 262056 78% 0.09K 8351 40 33404K buffer_head
- 151040 150537 99% 0.74K 30208 5 120832K ext3_inode_cache
It is found that most of the data (about 3.5 GB) is used for dentry_cache.
Problem solving
1. modify/proc/sys/vm/drop_caches to release the cache memory space occupied by Slab (refer to the official drop_caches documentation ):
- Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.
- To free pagecache:
- * echo 1 > /proc/sys/vm/drop_caches
- To free dentries and inodes:
- * echo 2 > /proc/sys/vm/drop_caches
- To free pagecache, dentries and inodes:
- * echo 3 > /proc/sys/vm/drop_caches
- As this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed.
- This tunable was added in 2.6.16.
2. Method 1 requires the user to have the root permission. if the user is not the root but has the sudo permission, you can use the sysctl command to set it:
- $sync
- $sudo sysctl -w vm.drop_caches=3
- $sudo sysctl -w vm.drop_caches=0 #recovery drop_caches
After the operation, you can use sudo sysctl-a | grep drop_caches to check whether the operation takes effect.
3. modify/proc/sys/vm/vfs_cache_pressure and adjust the priority (100 by default) for clearing inode/dentry caches. there are related explanations in LinuxInsight:
- At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a “fair” rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.
For specific setting methods, refer to Method 1 or method 2.
References
- Https://www.kernel.org/doc/Documentation/sysctl/vm.txt
- Http://major.io/2008/12/03/reducing-inode-and-dentry-caches-to-keep-oom-killer-at-bay/
- Http://linux-mm.org/Drop_Caches
The following records the progress of further troubleshooting.
Deeper reasons
I have found that there is a large amount of dentry_cache memory in Linux. why is there so much dentry_cache?
1. first, clarify the concept and function of dentry_cache: directory item high-speed cache is designed by Linux to improve the processing efficiency of directory item objects. it records the ing relationship between Directory items and inode. Therefore, when the application initiates a stat system call, it will create the corresponding dentry_cache item (further, if each stat file does not exist, there will always be a large number of new dentry_cache items created ).
2. the current server is a node of the storm cluster. first, I thought of storm-related working processes. strace the storm worker process found that there were very frequent stat system calls, in addition, the stat file is always a new file name:
sudo strace -fp
-e trace=stat
3. it is further observed that storm's worker process will frequently create, open, close, and delete heartbeat files in the local directory, with a new file name every second:
sudo strace -fp
-e trace=open,stat,close,unlink
The above is why there are so many dentry_caches in the system.
A strange phenomenon
By observing/proc/meminfo, we found that slab memory is divided into two parts:
SReclaimable // recoverable slabSUnreclaim // unrecoverable slab
At that time, the status of the server was: some of the memory occupied by slab, most of which were displayed as SReclaimable, that is, they could be recycled.
However, we can see through slabtop that the OBJS of dentry_cache, the most important part of slab memory, is almost ACTIVE, and 100% is in use.
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 13926348 13926348 100% 0.21K 773686 18 3494744K dentry_cache334040 262056 78% 0.09K 8351 40 33404K buffer_head151040 150537 99% 0.74K 30208 5 120832K ext3_inode_cache
Why is it recycled but ACTIVE? After seeing the Linux kernel, I am eager to explain :(
Will dcache not be automatically recycled and released due to its ACTIVE status?
Allow the system to automatically recycle dcache
As we mentioned in the previous section, most of the slab memory on the server is SReclaimable. can we give the OS a chance to automatically trigger the recycle operation at a certain time? The answer is yes.
I checked some information about Linux dcache and found that the operating system will trigger the kswapd kernel process to release it after the memory critical threshold is reached. the calculation method of this threshold is as follows:
1. first, grep low/proc/zoneinfo. the following result is obtained:
low 1 low 380 low 12067
2. this threshold is used to add up the above three columns and multiply them by 4 kB. after calculation, it is found that the recovery threshold of the current server is only 48 MB, so it is difficult to see this phenomenon, in reality, the operating system will fail to respond to hang if it does not wait for recovery.
3. you can use the following method to increase the threshold: vm. extra_free_kbytes is set to vm. min_free_kbytes is as large as it is, the low threshold in/proc/zoneinfo is doubled, and the high threshold is also increased, and so on.
$ sudo sysctl -a | grep free_kbytes vm.min_free_kbytes = 39847vm.extra_free_kbytes = 0
$ sudo sysctl -w vm.extra_free_kbytes=836787 ######1GB
4. for example, when the low threshold is set to 1 GB and the free memory of the system is smaller than 1 GB, it is observed that the kswapd process starts to work (the process state changes from Sleeping to Running ), at the same time, dcache starts to be recycled by the system until the free memory of the system is between the low threshold and the high threshold.
Link: http://www.cnblogs.com/panfeng412/p/drop-caches-under-linux-system.html
Http://www.cnblogs.com/panfeng412/p/drop-caches-under-linux-system-2.html
We recommend that you use Linux with a memory of more than 90%. Is it normal?