"Redis" about Redis data expiration policy

1. The expiration time of the key in Redis
Use EXPIRE key seconds command to set the data expiration time. Returning 1 indicates that the setting is successful, and returning 0 indicates that the key does not exist or the expiration time cannot be set successfully. After the expiration time is set on the key, the key will be automatically deleted after the specified number of seconds. The key assigned the expiration time is said to be unstable in Redis.

When the key is deleted by the DEL command or reset by the SET and GETSET commands, the expiration time associated with it will be cleared

127.0.0.1:6379> setex s 20 1
OK
127.0.0.1:6379> ttl s
(integer) 17
127.0.0.1:6379> setex s 200 1
OK
127.0.0.1:6379> ttl s
(integer) 195
127.0.0.1:6379> setrange s 3 100
(integer) 6
127.0.0.1:6379> ttl s
(integer) 152
127.0.0.1:6379> get s
"1 \ x00 \ x00100"
127.0.0.1:6379> ttl s
(integer) 108
127.0.0.1:6379> getset s 200
"1 \ x00 \ x00100"
127.0.0.1:6379> get s
"200"
127.0.0.1:6379> ttl s
(integer) -1
Use PERSIST to clear the expiration time

127.0.0.1:6379> setex s 100 test
OK
127.0.0.1:6379> get s
"test"
127.0.0.1:6379> ttl s
(integer) 94
127.0.0.1:6379> type s
string
127.0.0.1:6379> strlen s
(integer) 4
127.0.0.1:6379> persist s
(integer) 1
127.0.0.1:6379> ttl s
(integer) -1
127.0.0.1:6379> get s
"test"
Using rename only changed the key value

127.0.0.1:6379> expire s 200
(integer) 1
127.0.0.1:6379> ttl s
(integer) 198
127.0.0.1:6379> rename s ss
OK
127.0.0.1:6379> ttl ss
(integer) 187
127.0.0.1:6379> type ss
string
127.0.0.1:6379> get ss
"test"
Note: After Redis2.6, the accuracy of the expire can be controlled within 0 to 1 millisecond, and the key expiration information is stored in the form of an absolute Unix timestamp (redis2.6 is stored in millisecond-level precision), so when multiple servers synchronize, Be sure to synchronize the time of each server

2. Redis expired key deletion strategy
There are three ways to expire the Redis key:

Passive deletion: when reading / writing an expired key, a lazy deletion strategy will be triggered to directly delete the expired key
Active deletion: Because the lazy deletion strategy cannot guarantee that the cold data is deleted in time, Redis will periodically eliminate a batch of expired keys
When the currently used memory exceeds the maxmemory limit, an active cleanup strategy is triggered
Passive deletion
Only when the key is operated (such as GET), REDIS will passively check whether the key has expired, if it expires, delete it and return NIL.

1. This delete strategy is CPU-friendly, and the delete operation will only be performed when it has to. It will not waste unnecessary CPU time on other expire keys.

2. However, this strategy is not friendly to memory. A key has expired, but it will not be deleted before it is operated. If a large number of expired keys exist but are rarely accessed, it will cause a lot of wasted memory space. The expireIfNeeded (redisDb * db, robj * key) function is located in src / db.c.

/ * ------------------------------------------------ -----------------------------
 * Expires API
 * ------------------------------------------------- --------------------------- * /

int removeExpire (redisDb * db, robj * key) {
    / * An expire may only be removed if there is a corresponding entry in the
     * main dict. Otherwise, the key will never be freed. * /
    redisAssertWithInfo (NULL, key, dictFind (db-> dict, key-> ptr)! = NULL);
    return dictDelete (db-> expires, key-> ptr) == DICT_OK;
}

void setExpire (redisDb * db, robj * key, long long when) {
    dictEntry * kde, * de;

    / * Reuse the sds from the main dict in the expire dict * /
    kde = dictFind (db-> dict, key-> ptr);
    redisAssertWithInfo (NULL, key, kde! = NULL);
    de = dictReplaceRaw (db-> expires, dictGetKey (kde));
    dictSetSignedIntegerVal (de, when);
}

/ * Return the expire time of the specified key, or -1 if no expire
 * is associated with this key (i.e. the key is non volatile) * /
long long getExpire (redisDb * db, robj * key) {
    dictEntry * de;

    / * No expire? Return ASAP * /
    if (dictSize (db-> expires) == 0 ||
       (de = dictFind (db-> expires, key-> ptr)) == NULL) return -1;

    / * The entry was found in the expire dict, this means it should also
     * be present in the main dict (safety check). * /
    redisAssertWithInfo (NULL, key, dictFind (db-> dict, key-> ptr)! = NULL);
    return dictGetSignedIntegerVal (de);
}

/ * Propagate expires into slaves and the AOF file.
 * When a key expires in the master, a DEL operation for this key is sent
 * to all the slaves and the AOF file if enabled.
 *
 * This way the key expiry is centralized in one place, and since both
 * AOF and the master-> slave link guarantee operation ordering, everything
 * will be consistent even if we allow write operations against expiring
 * keys. * /
void propagateExpire (redisDb * db, robj * key) {
    robj * argv [2];

    argv [0] = shared.del;
    argv [1] = key;
    incrRefCount (argv [0]);
    incrRefCount (argv [1]);

    if (server.aof_state! = REDIS_AOF_OFF)
        feedAppendOnlyFile (server.delCommand, db-> id, argv, 2);
    replicationFeedSlaves (server.slaves, db-> id, argv, 2);

    decrRefCount (argv [0]);
    decrRefCount (argv [1]);
}

int expireIfNeeded (redisDb * db, robj * key) {
    mstime_t when = getExpire (db, key);
    mstime_t now;

    if (when <0) return 0; / * No expire for this key * /

    / * Do n‘t expire anything while loading. It will be done later. * /
    if (server.loading) return 0;

    / * If we are in the context of a Lua script, we claim that time is
     * blocked to when the Lua script started. This way a key can expire
     * only the first time it is accessed and not in the middle of the
     * script execution, making propagation to slaves / AOF consistent.
     * See issue # 1525 on Github for more information. * /
    now = server.lua_caller? server.lua_time_start: mstime ();

    / * If we are running in the context of a slave, return ASAP:
     * the slave key expiration is controlled by the master that will
     * send us synthesized DEL operations for expired keys.
     *
     * Still we try to return the right information to the caller,
     * that is, 0 if we think the key should be still valid, 1 if
     * we think the key is expired at this time. * /
    if (server.masterhost! = NULL) return now> when;

    / * Return when this key has not expired * /
    if (now <= when) return 0;

    / * Delete the key * /
    server.stat_expiredkeys ++;
    propagateExpire (db, key);
    notifyKeyspaceEvent (REDIS_NOTIFY_EXPIRED,
        "expired", key, db-> id);
    return dbDelete (db, key);
}

/ * ------------------------------------------------ -----------------------------
 * Expires Commands
 * ------------------------------------------------- --------------------------- * /

/ * This is the generic command implementation for EXPIRE, PEXPIRE, EXPIREAT
 * and PEXPIREAT. Because the commad second argument may be relative or absolute
 * the "basetime" argument is used to signal what the base time is (either 0
 * for * AT variants of the command, or the current time for relative expires).
 *
 * unit is either UNIT_SECONDS or UNIT_MILLISECONDS, and is only used for
 * the argv [2] parameter. The basetime is always specified in milliseconds. * /
void expireGenericCommand (redisClient * c, long long basetime, int unit) {
    robj * key = c-> argv [1], * param = c-> argv [2];
    long long when; / * unix time in milliseconds when the key will expire. * /

    if (getLongLongFromObjectOrReply (c, param, & when, NULL)! = REDIS_OK)
        return;

    if (unit == UNIT_SECONDS) when * = 1000;
    when + = basetime;

    / * No key, return zero. * /
    if (lookupKeyRead (c-> db, key) == NULL) {
        addReply (c, shared.czero);
        return;
    }

    / * EXPIRE with negative TTL, or EXPIREAT with a timestamp into the past
     * should never be executed as a DEL when load the AOF or in the context
     * of a slave instance.
     *
     * Instead we take the other branch of the IF statement setting an expire
     * (possibly in the past) and wait for an explicit DEL from the master. * /
    if (when <= mstime () &&! server.loading &&! server.masterhost) {
        robj * aux;

        redisAssertWithInfo (c, key, dbDelete (c-> db, key));
        server.dirty ++;

        / * Replicate / AOF this as an explicit DEL. * /
        aux = createStringObject ("DEL", 3);
        rewriteClientCommandVector (c, 2, aux, key);
        decrRefCount (aux);
        signalModifiedKey (c-> db, key);
        notifyKeyspaceEvent (REDIS_NOTIFY_GENERIC, "del", key, c-> db-> id);
        addReply (c, shared.cone);
        return;
    } else {
        setExpire (c-> db, key, when);
        addReply (c, shared.cone);
        signalModifiedKey (c-> db, key);
        notifyKeyspaceEvent (REDIS_NOTIFY_GENERIC, "expire", key, c-> db-> id);
        server.dirty ++;
        return;
    }
}

void expireCommand (redisClient * c) {
    expireGenericCommand (c, mstime (), UNIT_SECONDS);
}

void expireatCommand (redisClient * c) {
    expireGenericCommand (c, 0, UNIT_SECONDS);
}

void pexpireCommand (redisClient * c) {
    expireGenericCommand (c, mstime (), UNIT_MILLISECONDS);
}

void pexpireatCommand (redisClient * c) {
    expireGenericCommand (c, 0, UNIT_MILLISECONDS);
}

void ttlGenericCommand (redisClient * c, int output_ms) {
    long long expire, ttl = -1;

    / * If the key does not exist at all, return -2 * /
    if (lookupKeyRead (c-> db, c-> argv [1]) == NULL) {
        addReplyLongLong (c, -2);
        return;
    }
    / * The key exists. Return -1 if it has no expire, or the actual
     * TTL value otherwise. * /
    expire = getExpire (c-> db, c-> argv [1]);
    if (expire! = -1) {
        ttl = expire-mstime ();
        if (ttl <0) ttl = 0;
    }
    if (ttl == -1) {
        addReplyLongLong (c, -1);
    } else {
        addReplyLongLong (c, output_ms? ttl: ((ttl + 500) / 1000));
    }
}

void ttlCommand (redisClient * c) {
    ttlGenericCommand (c, 0);
}

void pttlCommand (redisClient * c) {
    ttlGenericCommand (c, 1);
}

void persistCommand (redisClient * c) {
    dictEntry * de;

    de = dictFind (c-> db-> dict, c-> argv [1]-> ptr);
    if (de == NULL) {
        addReply (c, shared.czero);
    } else {
        if (removeExpire (c-> db, c-> argv [1])) {
            addReply (c, shared.cone);
            server.dirty ++;
        } else {
            addReply (c, shared.czero);
        }
    }
}
But this is not enough, because there may be some keys that will never be accessed again. These keys with an expiration time need to be deleted after the expiration. We can even think of this situation as a kind of memory Leakage-useless garbage data takes up a lot of memory, but the server does not release them by itself. This is definitely not good news for Redis servers whose running state is very dependent on memory.

Actively delete
Let me talk about time events first. For a continuously running server, the server needs to regularly check and sort its own resources and status to maintain the server in a healthy and stable state. cron job)

In Redis, normal operations are implemented by redis.c / serverCron, which mainly performs the following operations

Update various statistics of the server, such as time, memory usage, database usage, etc.
Clean up expired key-value pairs in the database.
Resize the unreasonable database.
Close and clean up clients that have failed connections.
Try an AOF or RDB persistence operation.
If the server is the master node, periodically synchronize the slave nodes.
If you are in cluster mode, perform periodic synchronization and connection tests on the cluster.
Redis runs serverCron as a time event to ensure that it will automatically run every once in a while, and because serverCron needs to run regularly during Redis server operation, it is a cyclic time event: serverCron will continue to execute regularly until Until the server is shut down.

In Redis 2.6 version, the program stipulates that serverCron runs 10 times per second, with an average of every 100 milliseconds. Starting with Redis 2.8, users can adjust the number of serverCron executions per second by modifying the hz option. For details, please refer to the description of the hz option in the redis.conf file

 

It is also called timed deletion. The "periodic" here refers to the cleaning strategy triggered by Redis periodically, which is completed by the activeExpireCycle (void) function located in src / redis.c.

serverCron is a positioning task driven by redis' event framework. In this scheduled task, the activeExpireCycle function will be called. For each db, the expired key may be deleted more often within the limited time REDIS_EXPIRELOOKUPS_TIME_LIMIT. The reason for limiting the time is to prevent too long. The blocking affects the normal operation of redis. This active deletion strategy makes up for the memory-unfriendly nature of the passive deletion strategy.

Therefore, Redis will periodically randomly test a batch of keys with an expiration time and process them. The expired keys tested will be deleted. The typical way is that Redis does the following steps 10 times per second:

Random test 100 sets the expiration time Key
Delete all found expired keys
If more than 25 keys are deleted, repeat step 1
This is a simple algorithm based on probability. The basic assumption is that the extracted samples can represent the entire key space. Redis continues to clean up the expired data until the percentage of keys to be expired falls below 25%. This also means that the number of keys that have expired at any given moment but still occupy memory space is at most the number of write operations per second divided by 4.

The default value in Redis-3.0.0 is 10, which means that background tasks are called 10 times per second.

In addition to the frequency of active elimination, Redis also has a limit on the maximum length of time for each elimination task. This ensures that each active elimination will not block application requests too much. The following is the calculation formula for this limitation:

#define ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC 25 / * CPU max% for keys collection * /
...
timelimit = 1000000 * ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC / server.hz / 100;
Increasing hz will increase the frequency of Redis active elimination. If your Redis storage contains a lot of cold data and the memory is too large, you can consider increasing this value, but Redis authors recommend that this value should not exceed 100. We actually increased this value to 100 online, and observed that the CPU will increase by about 2%, but the memory release speed for cold data does increase significantly (by observing the number of keyspaces and used_memory size).

It can be seen that timelimit and server.hz are a reciprocal relationship, that is, the larger the hz configuration, the smaller the timelimit. In other words, the higher the expected frequency of active elimination per second, the shorter the maximum occupation time for each elimination. The maximum elimination time per second here is fixed at 250ms (1000000 * ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC / 100), and the elimination frequency and the maximum time for each elimination are controlled by the hz parameter.

From the above analysis, when the ratio of expired keys in redis does not exceed 25%, increasing the hz can significantly increase the minimum number of scan keys. Assuming that hz is 10, at least 200 keys are scanned in one second (10 calls per second * at least 20 keys are randomly selected each time), if hz is changed to 100, at least 2000 keys are scanned in one second; on the other hand If the expired key ratio exceeds 25%, there is no upper limit on the number of scan keys, but the CPU time takes up to 250ms per second.

When REDIS is running in the master-slave mode, only the master node will execute the above two expired deletion strategies, and then synchronize the delete operation "del key" to the slave node.

maxmemory
When the currently used memory exceeds the maxmemory limit, an active cleanup strategy is triggered

volatile-lru: LRU only for keys with an expiration time set (default value)
allkeys-lru: delete the key of the lru algorithm
volatile-random: randomly delete keys that are about to expire
allkeys-random: randomly delete
volatile-ttl: delete the expired
noeviction: never expires and returns an error when mem_used memory has exceeded the maxmemory setting. For all read and write requests, the redis.c / freeMemoryIfNeeded (void) function will be triggered to clear the excess memory. Note that this cleaning process is blocked until enough memory space is cleared. Therefore, if maxmemory is reached and the caller is still writing, the active cleanup strategy may be repeatedly triggered, resulting in a certain delay in the request.
When the mem_used memory has exceeded the maxmemory setting, for all read and write requests, redis.c / freeMemoryIfNeeded (void) function will be triggered to clear the excess memory. Note that this cleaning process is blocked until enough memory space is cleared. Therefore, if maxmemory is reached and the caller is still writing, the active cleanup strategy may be repeatedly triggered, resulting in a certain delay in the request.

When cleaning, the appropriate cleaning is performed according to the maxmemory-policy configured by the user (generally LRU or TTL). The LRU or TTL strategy here is not for all keys of redis, but the maxmemory-samples keys in the configuration file as The sample pool is cleaned by sampling.

The default configuration of maxmemory-samples in redis-3.0.0 is 5, if it increases, it will improve the accuracy of LRU or TTL. The result of redis author test is that when this configuration is 10, it is already very close to the accuracy of the full LRU And increasing maxmemory-samples will cause more CPU time to be consumed during active cleanup. Suggestions:

Try not to trigger maxmemory. It is best to adjust the hz to speed up elimination or cluster expansion after the mem_used memory usage reaches a certain percentage of maxmemory.
If you can control the memory, you don't need to modify the maxmemory-samples configuration; if Redis itself is used as an LRU cache service (this service is generally in the maxmemory state for a long time, and Redis will automatically do LRU elimination), you can adjust the maxmemory-samples appropriately.
The following is a description of the configuration parameters mentioned above

# Redis calls an internal function to perform many background tasks, like
# closing connections of clients in timeout, purging expired keys that are
# never requested, and so forth.
##
# Not all tasks are performed with the same frequency, but Redis checks for
# tasks to perform according to the specified "hz" value.
##
# By default "hz" is set to 10. Raising the value will use more CPU when
# Redis is idle, but at the same time will make Redis more responsive when
# there are many keys expiring at the same time, and timeouts may be
# handled with more precision.
##
# The range is between 1 and 500, however a value over 100 is usually not
# a good idea. Most users should use the default of 10 and raise this up to
# 100 only in environments where very low latency is required.
hz 10

# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
# is reached. You can select among five behaviors:
##
# volatile-lru-> remove the key with an expire set using an LRU algorithm
# allkeys-lru-> remove any key according to the LRU algorithm
# volatile-random-> remove a random key with an expire set
# allkeys-random-> remove a random key, any key
# volatile-ttl-> remove the key with the nearest expire time (minor TTL)
# noeviction-> do n‘t expire at all, just return an error on write operations
##
# Note: with any of the above policies, Redis will return an error on write
# operations, when there are no suitable keys for eviction.
##
# At the date of writing these commands are: set setnx setex append
# incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd
# sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby
# zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby
# getset mset msetnx exec sort
##
# The default is:
##
maxmemory-policy noeviction

# LRU and minimal TTL algorithms are not precise algorithms but approximated
# algorithms (in order to save memory), so you can tune it for speed or
# accuracy. For default Redis will check five keys and pick the one that was
# used less recently, you can change the sample size using the following
# configuration directive.
##
# The default of 5 produces good enough results. 10 Approximates very closely
# true LRU but costs a bit more CPU. 3 is very fast but not very accurate.
##
maxmemory-samples 5
Expiry processing in Replication link and AOF files
In order to obtain correct behavior without causing consistency problems, when a key expires, the DEL operation will be recorded in the AOF file and passed to all related slaves. That is, the expired deletion operation is carried out in the master instance and passed down, instead of being controlled by each salve. In this way, there will be no data inconsistency. When the slave is connected to the master, it cannot immediately clean up the expired key (need to wait for the DEL operation passed by the master). The slave still needs to manage and maintain the expired state in the data set so that the slave can be promoted to the master The expiration process is performed independently as well.

[Redis] About Redis data expiration strategy