The principle and realization mechanism of primary key failure in Redis _redis

As an important mechanism for periodically cleaning up invalid data, primary key failures exist in most caching systems, and Redis is no exception. Of the many commands provided by Redis, EXPIRE, Expireat, Pexpire, Pexpireat, and Setex and Psetex can be used to set the expiration time of a key-value pair, and a key-value pair is associated with the expiration time will automatically be deleted after expiration (or, more accurately, become inaccessible). It can be said that the concept of primary key failure is relatively easy to understand, but in the specific implementation to the Redis? Recently Benbow the main key failure mechanism in the Redis has a number of questions, and based on these questions on its careful exploration, is summarized as follows, to treat you spectators.

I. Control of failure time

In addition to invoking the persist command, is there anything else that will undo the expiration time of a primary key? The answer is yes. First, when you delete a primary key by using the DEL command, the expiration time will naturally be revoked (this is not nonsense, haha). Second, when a primary key that is set to fail time is overwritten, the expiration time of the primary key is also revoked (which seems to be nonsense, haha). But it should be noted that this is where the primary key is overwritten, but the value corresponding to the primary key is overwritten, so SET, Mset, or getset may cause the primary key to be overwritten, while the INCR, DECR, Lpush, Hset, and so on, are updated with the values of the primary key. This type of operation does not touch the expiration time of the primary key. In addition, there is a special command is RENAME, when we use RENAME to rename a primary key, the previously associated expiration time is automatically passed to the new primary key, but if a primary key is covered by RENAME (such as the primary key hello may be ordered RENAME World Hello), when the expiration time of the overridden primary key is automatically revoked, and the new primary key continues to retain the original primary key attribute.

Ii. internal implementation of failure

How does the primary key failure in Redis be implemented, that is, how the failed primary key is deleted? In fact, there are two main ways that Redis delete a failed primary key:

1. Negative method (passive way), when the primary key is accessed when it is found to be invalidated, then delete it
2. Active method (active way), periodically select a part of the failed primary key from the primary key that set the expiration time to delete

Internal representation of failure

Next, we will explore the implementation of these two methods through the code, but before we look at how the Redis management and maintenance of the primary key (note: The source code in this blog is all from Redis-2.6.12).

"Code snippet One" gives a definition of the structure of the database in Redis, which, in addition to the ID, is a pointer to a dictionary, where we look only at dict and expires, which maintain all key-value pairs contained in a REDIS database (the structure can be understood Is Dict[key]:value, that is, the mapping between the primary key and the value, and the latter is used to maintain a primary key (whose structure can be interpreted as expires[key]:timeout, a mapping of the primary key and the expiration time) that is set in the Redis database. When we insert data into the system using the Setex and Psetex commands, Redis first adds the key and Value to the dictionary table dict, and then adds the key and expiration time to the dictionary table of expires. When we use the EXPIRE, Expireat, Pexpire, and Pexpireat commands to set the expiration time of a primary key, Redis first go to dict this dictionary table to find out whether the primary key to be set exists, and if so, add the primary key and the expiration time to the expires This dictionary table. Simply put, the primary key and the specific expiration time that set the expiration time are all maintained in the Expires dictionary table.

"Code Snippet One":

Copy Code code as follows:

typedef struct REDISDB {
Dict *dict;
Dict *expires;
Dict *blocking_keys;
Dict *ready_keys;
Dict *watched_keys;
int id;
} Redisdb;

Negative approach

Having a rough idea of how redis is maintaining a primary key that has been set to fail time, let's take a look at how Redis is implementing a negative deletion of the failed primary key. "Code snippet Two" gives a function called expireifneeded, which is invoked in any function that accesses data, meaning that Redis is invoked when all commands involving reading data are implemented, such as GET, Mget, Hget, Lrange, and so on. The significance of its existence is to read the data before it has no expiration, if it fails to delete it. "Code snippet Two" gives all the relevant descriptions of the expireifneeded function, and it doesn't repeat its implementation here. Here is another function called in the expireifneeded function Propagateexpire, which is used to broadcast information that the primary key has failed before formally deleting the failed primary key, which is propagated to two destinations: one is sent to the AoF file, The operation to delete the failed primary key is recorded in the standard command format of Del Key, and the other is to send all the Slave to the current Redis server, which will also remove the invalid primary key from the operation in the standard command format of Del Key to notify these Slave to delete their respective failed primary keys. From this we can know that all the Redis servers that run as Slave do not need to use negative methods to remove the failed primary key, they only need to master to be obedient to OK!

"Code snippet Two":

Copy Code code as follows:

int expireifneeded (Redisdb *db, RobJ *key) {
Get the expiration time of a primary key
Long Long when = Getexpire (Db,key);
If the expiration time is negative, the primary key does not set the expiration time (the default is 1), direct return 0
if (when < 0) return 0;
If the Redis server is loading data from the Rdb file, temporarily does not fail the primary key deletion, returns directly 0
if (server.loading) return 0;
If the current Redis server is running as a slave, the deletion of the failed primary key is not performed because slave
The deletion of the failed primary key is controlled by master, but the expiration time of the primary key and the current time are
To tell the caller if the specified primary key is invalid.
if (server.masterhost!= NULL) {
return Mstime () > when;
}
If none of the above conditions are met, the failure time of the primary key is compared with the current time, if the specified primary key is found
return 0 directly without expiration
if (Mstime () <=) return 0;
If you find that the primary key is actually invalidated, first update the statistics on the failed primary key and then lose the primary key
Effective information, and finally remove the primary key from the database
server.stat_expiredkeys++;
Propagateexpire (Db,key);
Return Dbdelete (Db,key);
}

"Code snippet three":

Copy Code code as follows:

void Propagateexpire (Redisdb *db, RobJ *key) {
RobJ *argv[2];
Shared.del is a common Redis object that has been initialized at the beginning of the Redis server startup, that is, del command
Argv[0] = Shared.del;
ARGV[1] = key;
Incrrefcount (Argv[0]);
Incrrefcount (argv[1]);
Check if the Redis server is aof, and if it is turned on, log a del log for the failed primary key
if (server.aof_state!= redis_aof_off)
Feedappendonlyfile (server.delcommand,db->id,argv,2);
Check to see if the Redis server has slave, and if it is, send a command to all slave for the Del Fail primary key, which is
The reason why you do not have to actively delete the failed primary key when you find yourself in the expireifneeded function above is slave, because it
Just listen to the commands that Master sent over, OK?
if (Listlength (server.slaves))
Replicationfeedslaves (server.slaves,db->id,argv,2);
Decrrefcount (Argv[0]);
Decrrefcount (argv[1]);
}

Positive approach

The above is an introduction to the expireifneeded function to see how Redis removes the failed primary key in a negative way, but it is clearly not enough in this way, because if some of the failed primary keys are delayed to be accessed again, Redis You will never know that these primary keys have been invalidated and will never delete them, which would undoubtedly lead to a waste of memory space. As a result, Redis also prepares a positive deletion method, which uses Redis time events to implement a number of specified operations that are interrupted at intervals, including checking and deleting the failed primary key. Here we say the callback function for the time event is Servercron, which is created at Redis Server startup, and the number of executions per second is specified by the macro definition redis_default_hz, which is executed 10 times per second. Code snippet four gives the code for the program at the time the event was created, in the Initserver function of the redis.c file. In fact, Servercron this callback function not only for the failure of the primary key to check and delete, but also to update the statistics, client Connection timeout control, Bgsave and AOF trigger, etc., here we focus only on the deletion of the failure of the primary key implementation, that is, the function Activeexpirecycle.

"Code snippet Four":

Copy Code code as follows:

if (Aecreatetimeevent (Server.el, 1, servercron, NULL, NULL) = = Ae_err) {
Redispanic ("Create Time event failed");
Exit (1);
}

"Code snippet Five" gives the implementation of function activeexpirecycle and its detailed description, the main principle is to traverse the processing of Redis server in each database expires dictionary table, from which to try random sampling redis_expirelookups_per _cron (the default is 10) set the primary key of the expiration time, check whether they have been invalidated and delete the invalid primary key, if the number of primary key failure of the number of the sample is more than 25%,redis will think that the current database is still a lot of invalid primary key, So it will proceed to the next round of random sampling and deletion until just a fraction of 25% before stopping the processing of the current database and moving to the next database. What we need to be aware of here is that the Activeexpirecycle function does not attempt to process all the databases in Redis at once, but only to process Redis_dbcron_dbs_per_call (the default is 16), in addition The Activeexpirecycle function also has processing time limitations, not how long it takes to execute, all of which has only one purpose, and that is to avoid ineffective primary key deletion consuming too much CPU resources. "Code snippet Five" has a detailed description of all activeexpirecycle code, from which you can understand the specific implementation of the function.

"Code snippet Five":

Copy Code code as follows:

void Activeexpirecycle (void) {
Because every call to the Activeexpirecycle function does not check all the Redis databases at once, you need to record
The number of the last Redis database processed each time a function call, so that the next call to the Activeexpirecycle function
You can also proceed from this database, which is why current_db is declared static, and the other
Variable timelimit_exit is to record whether the last call to the Activeexpirecycle function was performed at
To time limit, so also need to declare as static
static unsigned int current_db = 0;
static int timelimit_exit = 0;
unsigned int J, iteration = 0;
The number of Redis databases processed by the Activeexpirecycle function is Redis_dbcron_dbs_per_call per call
unsigned int dbs_per_call = Redis_dbcron_dbs_per_call;
Long Long start = Ustime (), timelimit;
If the number of databases in the current Redis server is less than Redis_dbcron_dbs_per_call, all databases are processed.
If the last call to the Activeexpirecycle function has reached the time limit, it means that there are more invalid primary keys and
Will choose to process all databases
if (Dbs_per_call > Server.dbnum | | timelimit_exit)
Dbs_per_call = Server.dbnum;
The maximum time (in microseconds) to perform the Activeexpirecycle function, where REDIS_EXPIRELOOKUPS_TIME_PERC
Is the percentage of CPU time that can be allocated to the Activeexpirecycle function within a unit of time, and the default value is 25,server.hz
That is the number of activeexpirecycle calls in a second, so this formula is more clearly written in this way, that is
(1000000 * (redis_expirelookups_time_perc/100))/server.hz
TimeLimit = 1000000*redis_expirelookups_time_perc/server.hz/100;
Timelimit_exit = 0;
if (timelimit <= 0) timelimit = 1;
Traverse processing of failed data in each Redis database
for (j = 0; J < Dbs_per_call; J + +) {
int expired;
Redisdb *db = server.db+ (current_db% server.dbnum);
The current_db will be added one at a time to ensure that even if you do not delete all current
Invalid primary key in the database, and the next call to Activeexpirecycle will start processing from the next database.
So that every database has a chance to be processed.
current_db++;
Start processing the failed primary key in the current database
do {
unsigned long num, slots;
Long Long now;
If the Expires dictionary table size is 0, there is no primary key to set the expiration time in the database, check directly
A database
if (num = dictsize (db->expires)) = = 0) break;
Slots = Dictslots (db->expires);
now = Mstime ();
If the Expires dictionary table is not empty, but its padding rate is less than 1%, then the cost of randomly selecting a primary key for checking
It's going to be high, so check the next database directly.
if (num && slots > Dict_ht_initial_size &&
(Num*100/slots < 1)) Break
expired = 0;
If the number of entry in the Expires dictionary table is insufficient to reach the number of samples, select all key as Sample
if (num > Redis_expirelookups_per_cron)
num = Redis_expirelookups_per_cron;
while (num--) {
Dictentry *de;
Long Long T;
Randomly obtain a primary key that sets the expiration time to check if it has been invalidated
if (de = Dictgetrandomkey (db->expires)) = = NULL) break;
t = Dictgetsignedintegerval (DE);
if (now > t) {
It was found that the primary key was actually invalidated and the primary key was deleted
SDS key = Dictgetkey (DE);
RobJ *keyobj = Createstringobject (Key,sdslen (key));
Also broadcast the failure information for the primary key before deleting it
Propagateexpire (Db,keyobj);
Dbdelete (Db,keyobj);
Decrrefcount (Keyobj);
expired++;
server.stat_expiredkeys++;
}
}
After each sample deletion of iteration plus one, every 16 times after the sample deleted check whether this time of execution
The time limit has been reached, and if the time limit has been reached, record this execution reaches the time limit and exits
iteration++;
if ((iteration & 0xf) = = 0 &&
(Ustime ()-start) > TimeLimit)
{
Timelimit_exit = 1;
Return
}
Continue the sample deletion process if the number of primary keys that are invalidated is greater than 25% of the number of samples
while (Expired > REDIS_EXPIRELOOKUPS_PER_CRON/4);
}
}

Third, Memcached delete the failure of the primary key method and Redis what is the similarities and differences?

First, Memcached is also a negative way to remove a failed primary key, that is, the Memcached does not monitor whether the primary key is invalidated, but only if the primary key is accessed through a get. Second, the biggest difference between Memcached and Redis in the primary key failure mechanism is that Memcached does not really remove the failed primary key as Redis does, but simply reclaims the space occupied by the failed primary key. Thus, when new data is written to the system, Memcached takes precedence over the space of those failed primary keys. If the space for the failed primary key is used up, Memcached can also reclaim the space that has not been accessed for a long time through the LRU mechanism, so Memcached does not need a periodic delete operation like Redis, which is also determined by the memory management mechanism used by Memcached. At the same time, it should be pointed out that redis in the presence of Oom can also be configured maxmemory-policy This parameter to determine whether to use the LRU mechanism to reclaim the memory space (thanks to @jonathan_dai classmate in the Redis LRU mechanism of the original text of correction). In Redis, LRU is the default mechanism, and you might ask, if all keys are not set to expire, and Redis memory footprint is maxmemory, what happens when you add or modify keys? If there is no suitable key to remove, Redis will return an error when it is written. See Redis configuration files based on version 2.8

Redis failure mechanism of primary key will affect system performance?

Through the introduction of the Redis primary key failure mechanism, we know that although Redis will periodically check the primary key that set the expiration time and delete the failed primary key, but by limiting the number of databases per processing, the Activeexpirecycle function in one second, The limitation of the CPU time allocated to the Activeexpirecycle function, the limit of the number of failed primary keys that continues to be deleted, Redis has greatly reduced the effect of the primary key failure mechanism on the overall performance of the system, However, if there is a large number of primary keys in the actual application in a short period of time will also reduce the ability to respond to the system, so this situation should undoubtedly be avoided.