Track record of a large number of dirty page problems with Redis

Fromhttps://www.zybuluo.com/SailorXiao/note/136014 

Case scene

The line found a machine memory load is heavy, top found a redis process accounted for a large amount of memory, top content is as follows:

27190 Root 0 18.6g 18g S 0.3 59.2 926:17.83 redis-server  

I found that Redis accounted for 18.6G of physical memory. Since Redis is only used to cache some program data, it is inconceivable that the Redis Info command, found that the actual data occupy only 112M, as follows:

# Memory used_memory:118140384 used_memory_human:112.67m used_memory_rss:19903766528 used_memory_peak:17871578336 used_memory_peak_human:16.64g used_memory_lua:31744 mem_fragmentation_ratio:168.48 mem_allocator:libc  

so I used pmap-x 27190 to view the memory image information for the Redis process, the results are as follows:

27190:./redis-server. /redis.conf Address kbytes RSS Dirty Mode Mapping 0000000000400000 548 18 4 0 r-x--redis-server 0000000000689000-RW---redis-s erver 000000000068d000---[anon] 0000000001eb6000 132 132 ---[anon] 0000000001ed7000 19436648 19435752 19435752 rw---[anon] 00007f5 862cb2000 4 0 0-----[anon] 

There is a large amount of memory dirty pages found. The problem with high memory load is now clearer because Redis's dirty pages are taking up a lot of memory. But why are there so many dirty pages in Redis?

Case analysis

Look at the definition of a Linux dirty page:

 Dirty page is the concept of the Linux kernel, because the hard disk read and write faster than the speed of memory, the system will read and write more frequent data in advance in memory to improve read and write speed, this is called Cache, Linux is a page as a cache unit, when the process modifies the data in the cache, The page is marked as dirty by the kernel, and the kernel will write the dirty page data to disk at the appropriate time to keep the data in the cache consistent with the data in the disk. 

http://blog.chinaunix.net/uid-17196076-id-2817733.html 
Found flush with memory can be set (under/PROC/SYS/VM)

dirty_background_bytes/dirty_background_ratio:-When the dirty page of the system reaches a certain value (bytes or scale), the background process is started to brush the dirty page to disk, which does not affect the memory read/write (when the bytes setting exists, Ratio is automatically calculated) Dirty_bytes/dirty_ratio:-When the system's dirty page reaches a certain value (bytes or scale), the boot process brushes the dirty page to disk, where the memory write may be blocked (when the bytes setting is present, Ratio is automatically calculated) Dirty_expire_centisecs:-When the memory and the data in the disk inconsistencies exist (in units of 1% seconds), it is defined as dirty and is cleaned up in the next flush. Inconsistencies are based on the inode timestamp of the file on disk Dirty_writeback_centisecs:-The system will flush the dirty to disk (in 1% seconds) every time,  

Check the flush configuration of the current system, and find no problem, Dirty_background_ratio is 10%,dirty_ratio for 20%,dirty_writeback_centisecs for 5s,dirty_expire_ Centisecs is 30s, but why is Redis's dirty pages not flush to disk?

The general dirty page is to flush the in-memory data to the disk, then is the Redis persistence caused the dirty page? Review the following Redis configuration for these areas:

RDB persistence has been shut down # save 1 # Save # Save 10000 # Append persistence is also turned off AppendOnly no # maximum memory settings, memory substitution policies are default values # MaxMemory & Lt;bytes> # maxmemory-policy Volatile-lru  

VOLATILE-LRU: Pick the least Recently used data elimination (default) from the set expiration data set (Server.db[i].expires) Volatile-ttl: The dataset from which the expiration time has been set (server.db[i].exp IRES) Select the data that will expire Volatile-random: Choose any data from the set of expired Datasets (Server.db[i].expires) to retire Allkeys-lru: from the DataSet (server.db[i].dict Select the least recently used data to retire Allkeys-random: Choose any data from the dataset (Server.db[i].dict) to retire No-enviction: Prohibit eviction of data  

In the current usage environment, the program uses Redis as the cache, and the data is set to the expire timeout, waiting for Redis to be deleted after the expiration date. So dirty page reason, is not because of outdated data cleanup mechanism problems (such as cleaning is not timely, etc.)? Therefore, you need to look at the policies that Redis takes when deleting outdated data, and the reference information is as follows:
Memory release and expiration key deletion in Redis
Removal of the Redis expiration key

Redis expiration key removal mechanism:

 lazy Delete:-After the expire key expires, it is not automatically deleted, but checks every time the key is read, checks whether the key has expired, and deletes the action if it expires. This ensures that the delete operation will be deleted on a regular basis only if it is not possible:-cron deletes are performed at intervals, and a certain number of keys (20 by default) are randomly identified from the expire-keys of each db to check if the key has timed out.  If it has timed out, delete-by limiting the length and frequency of the delete operation, and to reduce the effect of the delete operation on the CPU time. Automatic deletion greater than maxmemory:-Each time the client and Server command interact, the server checks the usage of maxmemory-if the current memory exceeds max-memory, it is cleaned up until the memory occupies the M Axmemory line below-cleanup strategy based on elimination mechanism (lru,ttl,random, etc.)  

Redis for expired key deletion uses the policy of lazy Delete + periodic delete + more than MaxMemory Auto Purge.

Case Locator

Through the above analysis, the problem has been relatively clear, for the following reasons:

1. For some reason, Redis uses more and more memory (possibly due to lazy deletion, resulting in more data expire, or other reasons, depending on the implementation within Redis)
2. Redis does not actually read and write files because it is only a cache, so the operating system does not flush it to disk (because there is no place to flush)
3. Because Redis does not have a maxmemory set, the default is the machine's memory size and Redis cleans itself (VOLATILE-LRU mechanism) only if the memory used by Redis itself reaches the machine memory size
4. So the current Redis memory is getting bigger, and the dirty pages are getting more and more (most of them may be data that has expired)

Case resolution

In order to solve this problem, the more convenient and reasonable way is:

• Lazy deletion is sometimes not very reliable, especially for some log types of data, after writing to Redis, regardless of, although set the timeout period, but no effect
• Regular deletion is also not very reliable for expire-keys, there is randomness, it may also expire long data is not deleted
• Comparatively, a reasonable way is to set the maxmemory size of Redis based on usage to enable Redis to implement its own data cleansing mechanism to ensure that MEM is limited to the MaxMemory set range

Track record of a large number of dirty page problems with Redis