Detailed data expiration policy in Redis

I believe that we have a little knowledge of data expiration in Redis, this article mainly introduces the data expiration policy in Redis, the article introduces in detail through the example code, I believe that everyone's understanding and learning has a certain reference value, the need for friends can reference, hope to help everyone.

1, the Expiration time of key in Redis

Set the data expiration time by expire the key seconds command. A return of 1 indicates that the setting was successful, and returning 0 indicates that the key does not exist or the expiration is not successfully set. After the expiration time is set on the key, the key will be automatically deleted after the specified number of seconds. Keys that have been assigned an expiration time are known to be unstable in Redis.

The expiration time associated with a key being deleted by the del command or reset by the set, Getset command is cleared

127.0.0.1:6379> Setex s 1ok127.0.0.1:6379> ttl s (integer) 17127.0.0.1:6379> Setex s 1ok127.0.0.1:6379> TTL s (integer) 195127.0.0.1:6379> setrange s 3 (integer) 6127.0.0.1:6379> ttl s (integer) 152127.0.0.1:6379> g ET S "1\x00\x00100" 127.0.0.1:6379> ttl s (integer) 108127.0.0.1:6379> getset S "1\x00\x00100" 127.0.0.1:6379 > Get S "127.0.0.1:6379>" ttl s (integer)-1

Use persist to clear expiration time

127.0.0.1:6379> Setex s testok127.0.0.1:6379> get S "Test" 127.0.0.1:6379> ttl s (integer) 94127.0.0.1:6379 > Type sstring127.0.0.1:6379> strlen S (integer) 4127.0.0.1:6379> persist S (integer) 1127.0.0.1:6379> ttl s ( Integer) -1127.0.0.1:6379> get S "Test"

Using rename just changed the key value

127.0.0.1:6379> expire S (integer) 1127.0.0.1:6379> ttl s (integer) 198127.0.0.1:6379> rename s ssOK127.0.0.1 :6379> ttl SS (integer) 187127.0.0.1:6379> type ssstring127.0.0.1:6379> get ss "test"

Description: After Redis2.6 the expire accuracy can be controlled in 0 to 1 milliseconds, the expiration information of key is stored in the form of absolute Unix timestamp (after Redis2.6 is stored in millisecond-level precision), so it is important to synchronize the time of each server when multiple servers are synchronized.

2. Redis Expiration Key Delete policy

There are three ways that Redis key expires:

1. Passive deletion: When reading/writing a key that has expired, the lazy delete policy is triggered, and the expired key is deleted directly

2. Active deletion: Due to the lazy deletion policy cannot guarantee that cold data is deleted in time, so Redis will periodically proactively eliminate a batch of expired key

3. Active cleanup policy is triggered when current memory exceeds maxmemory limit
   

Passive deletion

Only when a key is manipulated (such as get) does Redis passively check if the key is out of date, delete it if it expires, and return nil.

1, this deletion policy is friendly to the CPU, the deletion operation is only in the case of having to do, will not waste unnecessary CPU time on other expire key.

2, but this policy is not friendly to memory, a key has expired, but it will not be deleted before it is manipulated, still occupy the memory space. If you have a large number of expired keys but are rarely accessed, it can cause a lot of wasted memory space. The expireifneeded (Redisdb *db, RobJ *key) function is located in SRC/DB.C.

/*-----------------------------------------------------------------------------* Expires API *------------------- ---------------------------------------------------------*/int Removeexpire (Redisdb *db, RobJ *key) {/* an expire may O nly be removed if there are a corresponding entry in the * main dict. Otherwise, the key would never be freed. */Redisassertwithinfo (Null,key,dictfind (db->dict,key->ptr) = NULL); Return Dictdelete (db->expires,key->ptr) = = DICT_OK;}  void Setexpire (Redisdb *db, RobJ *key, Long Long when) {Dictentry *kde, *de; /* Reuse the SDS from the main dict in the Expire dict */KDE = Dictfind (db->dict,key->ptr); Redisassertwithinfo (Null,key,kde! = NULL); de = Dictreplaceraw (Db->expires,dictgetkey (KDE)); Dictsetsignedintegerval (De,when);} /* Return The expire time of the specified key, or-1 if no expire * is associated with this key (i.e. the key is non Vola  Tile) */long long Getexpire (Redisdb *db, RobJ *key) {dictentry *de; /* No expire? Return ASAP */if (dictsize (db->expires) = = 0 | |  (de = Dictfind (db->expires,key->ptr)) = = NULL) return-1; /* The entry is found in the expire dict, this means it should also * is present in the main dict (Safety Check). */Redisassertwithinfo (Null,key,dictfind (db->dict,key->ptr) = NULL); Return Dictgetsignedintegerval (DE);} /* Propagate expires into slaves and the AOF file. * When a key expires in the master, a DEL operation for this key was sent * to all the slaves and the AOF file if enabled. * * This "is" the key expiry is centralized in one place, and since both * AOF and the Master->slave link guarantee oper ation ordering, everything * 'll be consistent even if we allow write operations against expiring * keys.  */void Propagateexpire (Redisdb *db, RobJ *key) {robj *argv[2]; Argv[0] = Shared.del; ARGV[1] = key; Incrrefcount (Argv[0]);  Incrrefcount (argv[1]); if (server.aof_state! = Redis_aof_off) feedappendonlyfile (server.delcommand,db->id,argv,2); ReplicationfeedSlaves (server.slaves,db->id,argv,2); Decrrefcount (Argv[0]); Decrrefcount (argv[1]);}  int expireifneeded (Redisdb *db, RobJ *key) {mstime_t when = Getexpire (Db,key); mstime_t now; if (when < 0) return 0; /* No expire for the This key *//* Don ' t expire anything while loading. It'll be done later. */if (server.loading) return 0; /* If We are in the context of a LUA script, we claim that time was * blocked to when the Lua script started. This-a key can expire * Only the first time it was accessed and not in the middle of the * script execution, making pro Pagation to slaves/aof consistent. * See issue #1525 on Github for more information. * * now = Server.lua_caller? Server.lua_time_start:mstime (); /* If We are running in the context of a Slave, return ASAP: * The slave key expiration was controlled by the master that W Ill * send us synthesized DEL operations for expired keys. * * Still we try to return the right information to the caller, * This is, 0 if we think the key should Be still valid, 1 if * We think the key was expired at this time.  */if (server.masterhost! = NULL) return now > when; /* Return when the This key had not expired */if (now <= when) return 0; /* Delete the key */server.stat_expiredkeys++; Propagateexpire (Db,key); Notifykeyspaceevent (redis_notify_expired, "EXPIRED", key,db->id); Return Dbdelete (Db,key);} /*-----------------------------------------------------------------------------* Expires Commands *-------------- --------------------------------------------------------------*/* The generic command implementation for EXPIRE, Pexpire, Expireat * and Pexpireat. Because the Commad second argument may be relative or absolute * the "basetime" argument are used to signal what the base t IME is (either 0 * for *at variants of the command, or the current time for relative expires). * unit is either unit_seconds or unit_milliseconds, and are only used for * the argv[2] parameter. The basetime is always specified in milliseconds. */void Expiregenericcommand (redisclient *c, Long long basetime, int unit) {RobJ *key = c->argv[1], *param = c->argv [2]; Long long when; /* Unix time in milliseconds when the key is expire.  */if (getlonglongfromobjectorreply (c, Param, &when, NULL)! = REDIS_OK) return; if (unit = = unit_seconds) when *= 1000;  When + = Basetime; /* No key, return zero.  */if (Lookupkeyread (c->db,key) = = NULL) {addreply (C,shared.czero); return;} /* EXPIRE with negative TTL, or expireat with a timestamp into the past * should never is executed as a DEL when load the AOF or in the context * of a Slave instance. * * Instead We take the other branch of the IF statement setting a expire * (possibly in the past) and wait for a explic It DEL from the master. */if (when <= mstime () &&!server.loading &&!server.masterhost) {robj *aux; Redisassertwithinfo (C,ke Y,dbdelete (C->db,key));  server.dirty++; /* Replicate/aof this as an explicit DEL. */aux = Createstringobject("DEL", 3); Rewriteclientcommandvector (C,2,aux,key); Decrrefcount (aux); Signalmodifiedkey (C->db,key); Notifykeyspaceevent (Redis_notify_generic, "Del", Key,c->db->id); Addreply (c, Shared.cone); Return } else {setexpire (c->db,key,when); addreply (C,shared.cone); Signalmodifiedkey (C->db,key); notifyKeyspaceEvent (Redis_notify_generic, "expire", key,c->db->id); server.dirty++; Return }} void Expirecommand (Redisclient *c) {Expiregenericcommand (C,mstime (), unit_seconds);} void Expireatcommand ( Redisclient *c) {Expiregenericcommand (c,0,unit_seconds);} void Pexpirecommand (Redisclient *c) {ExpireGenericCommand ( C,mstime (), unit_milliseconds);} void Pexpireatcommand (Redisclient *c) {Expiregenericcommand (c,0,unit_milliseconds);} void Ttlgenericcommand (  Redisclient *c, int output_ms) {Long long expire, TTL =-1; /* If the key does not exist @ all, return-2 */if (Lookupkeyread (c->db,c->argv[1]) = = NULL) {Addreplylonglong (c, -2); Return }/* The key exists. Return-1 If IT has no expire, or the actual * TTL value otherwise. */expire = Getexpire (c->db,c->argv[1]); if (expire! =-1) {ttl = Expire-mstime (), if (TTL < 0) ttl = 0;} if (ttl = =-1) {Addreplylonglong (c,-1);} else {ad Dreplylonglong (C,output_ms ttl: ((ttl+500)/1000)); }} void Ttlcommand (Redisclient *c) {Ttlgenericcommand (c, 0);} void Pttlcommand (Redisclient *c) {Ttlgenericcommand (c, 1 ); } void Persistcommand (Redisclient *c) {dictentry *de; de = Dictfind (c->db->dict,c->argv[1]->ptr); if (de = = N  ULL) {addreply (C,shared.czero);} else {if (Removeexpire (c->db,c->argv[1])) {addreply (c,shared.cone); server.dirty++; } else {addreply (C,shared.czero);}}}

But this is not enough, because there may be some key will never be accessed again, these set the expiration time of the key also need to be deleted after expiration, we can even consider this situation as a memory leak----useless garbage data consumes a lot of memory, The server does not release them on its own, which is certainly not good news for Redis servers that are very dependent on memory for running state.

Active deletion

Let's talk about time events, for servers that continue to run, the server needs to periodically check and defragment its own resources and state to keep the server in a healthy and stable state, which is collectively referred to as General operations (cron Job)

In Redis, general operations are implemented by Redis.c/servercron, which primarily performs the following actions

• Update the server's various statistics, such as time, memory usage, database occupancy, etc.

• Cleans up expired key-value pairs in the database.

• Sizing an unreasonable database.

• Shuts down and cleans up clients that fail the connection.

• Try a AOF or RDB persistence operation.

• If the server is the primary node, the subordinate nodes are synchronized periodically.

• If in cluster mode, the cluster is regularly synchronized and connected to the test.

Redis runs Servercron as a time event to ensure that it runs automatically at regular intervals, and because Servercron needs to run regularly during the Redis server run time, it is a recurring event: Servercron Until the server shuts down.

In the Redis 2.6 release, the program specifies that Servercron runs 10 times per second, averaging every 100 milliseconds. Starting with Redis 2.8, the user can modify the Hz option to adjust the number of executions per second of the Servercron, and refer to the redis.conf file for a description of the HZ option

Also called time-lapse, where "regular" refers to the cleanup policy that Redis periodically triggers, which is done by the activeexpirecycle (void) function located in SRC/REDIS.C.

Servercron is a positioning task driven by the Redis event framework, which calls the Activeexpirecycle function for each db at a limited time redis_expirelookups_time_ Limit the number of expired keys that may be removed later, the reason for limiting time is to prevent excessive blocking affecting the normal operation of Redis. This proactive deletion policy compensates for the unfriendly memory of the passive deletion policy.

As a result, redis periodically randomly tests a batch of keys that have an expiration time set and are processed. The expired key that was tested is deleted.

In a typical way, Redis does the following steps 10 times per second:

• Random test 100 keys with expiration time set

• Delete all discovered expired keys

• Repeat step 1 if you have deleted more than 25 keys
  

This is a simple algorithm based on probability, the basic assumption is that the extracted sample can represent the entire key space, and Redis continuously cleans up the expired data until the percentage of the key that will expire drops below 25%. This also means that the amount of the key that has expired at any given moment but still occupies the memory space is up to the write operation amount per second divided by 4.

The default value in Redis-3.0.0 is 10, which represents a 10-time background task call per second.

In addition to the frequency of active elimination, Redis has a limit on the maximum length of time to perform each phase-out task, which ensures that the application request is not blocked too much for each active elimination, and the following is the qualifying calculation formula:

#define ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/* CPU max% for keys collection */... timelimit = 1000000*active_expire_cy cle_slow_time_perc/server.hz/100;

The Hz tune-up will increase the frequency of Redis proactive elimination, and if your redis storage contains too much cold data to take up too much memory, consider increasing the value, but the Redis authors recommend that this value not exceed 100. We actually set this value to 100 on the line and observed that the CPU will increase by about 2%, but the memory release rate of the cold data is obviously improved (by observing the number of keyspace and used_memory size).

It can be seen that timelimit and server.hz are a reciprocal relationship, that is, the greater the Hz configuration, the smaller the timelimit. In other words, the higher the frequency of proactive elimination per second is expected, the shorter the maximum time taken for each elimination. The maximum elimination time per second is fixed at 250ms (1000000*active_expire_cycle_slow_time_perc/100), while the elimination frequency and the maximum time of each elimination are controlled by the Hz parameter.

From the above analysis, when the expired key ratio in Redis does not exceed 25%, the increase in Hz can significantly increase the minimum number of scan keys. Assuming a Hz of 10, a minimum of 200 keys is scanned in one second (10 times per second * At least 20 keys at a time), and if Hz is changed to 100, a minimum of 2000 keys is scanned within one second, and if the expiration key ratio exceeds 25%, there is no limit to the number of scan keys. But CPU time consumes up to 250ms per second.

When Redis is running in master-slave mode, only the primary node will perform both of these expiration delete policies, and then synchronize the delete operation "Del key" to the slave node.

MaxMemory

Active cleanup policy is triggered when current memory exceeds maxmemory limit

• VOLATILE-LRU: LRU (default) only for keys that have an expiration time set

• ALLKEYS-LRU: Delete the LRU algorithm key

• Volatile-random: Randomly delete expiring key

• Allkeys-random: Random Delete

• Volatile-ttl: Delete the expiring

• Noeviction: Never expires, returns an error when mem_used memory has exceeded the maxmemory setting, the redis.c/freememoryifneeded (void) function is triggered for all read and write requests to clean up excess memory. Note that this cleanup process is blocked until enough memory space is cleared. So if the maxmemory is reached and the caller is still writing, the active cleanup policy may be triggered repeatedly, resulting in a delay in the request.

When the mem_used memory has exceeded the maxmemory setting, the redis.c/freememoryifneeded (void) function is triggered for all read and write requests to clean up the excess memory. Note that this cleanup process is blocked until enough memory space is cleared. So if the maxmemory is reached and the caller is still writing, the active cleanup policy may be triggered repeatedly, resulting in a delay in the request.

Cleanup will be based on user-configured Maxmemory-policy to do the appropriate cleanup (usually LRU or TTL), where the LRU or TTL policy is not for all of the Redis key, Instead, the Maxmemory-samples key in the configuration file is sampled as a sample pool.

The default configuration of the Maxmemory-samples in redis-3.0.0 is 5, and if increased, the accuracy of the LRU or TTL is improved, the result of the Redis author test is that when this configuration is 10 it is already very close to the full-volume LRU accuracy, and increasing the maxmemory-samples will cause Consume more CPU time during active cleanup, it is recommended that:

• Try not to trigger the maxmemory, preferably after the mem_used memory occupies a certain proportion of maxmemory, you need to consider the large Hz to speed up the elimination, or cluster expansion.

• If you can control the memory, you can not modify the Maxmemory-samples configuration, if Redis itself as the LRU cache service (this service is generally in the maxmemory state for a long time, by the Redis automatic LRU elimination), Can be suitably adjusted large maxmemory-samples.
  

The following is a description of the configuration parameters mentioned above

# Redis calls an internal function to perform many background tasks, like # Closing connections of clients in timeout, pur Ging expired keys that is # never requested, and so forth. # Not all tasks is performed with the same frequency, but Redis checks for # Tasks to perform according to the Specifie D "Hz" value. # # By default ' Hz ' is set to 10. Raising the value would use more CPU time # Redis was idle, but at the same time would make Redis + responsive when # ther E is many keys expiring at the same time, and timeouts May is # handled with more precision. #: The range is between 1 and, however a value of over are usually not # a good idea. Most users should use the default of ten and raise this up to # (environments) where very low latency is required . Hz # MaxMemory Policy:how Redis would select what to remove when MaxMemory # is reached. You can select among five behaviors: # # VOLATILE-LRU, remove the key with an expire set using a LRU algorithm # all KeYS-LRU, remove any key according to the LRU algorithm # volatile-random, remove a random key with an expire set # allkeys-random, remove a random key, any key # Volatile-ttl, remove the key with the nearest expire time (Mino R TTL) # Noeviction, expire at all, just return a error on write operations # # Note:with any of the above PO Licies, Redis would return an error in write # operations, when there is no suitable keys for eviction. # at the date of writing these commands Are:set setnx setex Append # incr decr rpush lpush rpushx lpushx linsert LSet R poplpush sadd # sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby # zunionstore zinterstore hset hsetnx Hmset Hincrby Incrby Decrby # getset mset msetnx exec Sort # # The default is: # maxmemory-policy noeviction # LRU and Mi Nimal TTL algorithms is not precise algorithms but approximated # algorithms (in order to save memory), so can tune I T-speed or # accuracy. For default RedIS would check five keys and pick the one that were # used less recently, you can change the sample size using the following # configuration directive. # # The default of 5 produces good enough results. Ten approximates very closely # true LRU but costs a bit more CPU. 3 is very fast and not very accurate. # maxmemory-samples 5

Replication expiration processing in link and aof files

In order to get the correct behavior without causing a consistency problem, when a key expires, the Del operation is recorded in the AoF file and passed to all related slave. The expiration delete operation is uniformly carried out in the master instance and passed down, rather than each salve in its own hands. This will not result in inconsistent data. When the slave is connected to master, it does not immediately clean up the expired key (which needs to wait for the Del operation passed by master), Slave still needs to manage the expiration status in the data set to make it easier for slave to be promoted to master, which can be processed as independently as master.