A Free Trial That Lets You Build Big!
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
As an important mechanism for periodically cleaning up invalid data, primary key failures exist in most caching systems, and Redis is no exception. Of the many commands provided by Redis, EXPIRE, Expireat, Pexpire, Pexpireat, and Setex and Psetex can be used to set the expiration time of a key-value pair, and a key-value pair is associated with the expiration time will automatically be deleted after expiration (or, more accurately, become inaccessible). It can be said that the concept of primary key failure is relatively easy to understand, but in the specific implementation to the Redis? Recently Benbow the main key failure mechanism in the Redis has a number of questions, and based on these questions on its careful exploration, is summarized as follows, to treat you spectators.
I. Control of failure time
In addition to invoking the persist command, is there anything else that will undo the expiration time of a primary key? The answer is yes. First, when you delete a primary key by using the DEL command, the expiration time will naturally be revoked (this is not nonsense, haha). Second, when a primary key that is set to fail time is overwritten, the expiration time of the primary key is also revoked (which seems to be nonsense, haha). But it should be noted that this is where the primary key is overwritten, but the value corresponding to the primary key is overwritten, so SET, Mset, or getset may cause the primary key to be overwritten, while the INCR, DECR, Lpush, Hset, and so on, are updated with the values of the primary key. This type of operation does not touch the expiration time of the primary key. In addition, there is a special command is RENAME, when we use RENAME to rename a primary key, the previously associated expiration time is automatically passed to the new primary key, but if a primary key is covered by RENAME (such as the primary key hello may be ordered RENAME World Hello), when the expiration time of the overridden primary key is automatically revoked, and the new primary key continues to retain the original primary key attribute.
Ii. internal implementation of failure
How does the primary key failure in Redis be implemented, that is, how the failed primary key is deleted? In fact, there are two main ways that Redis delete a failed primary key:
1. Negative method (passive way), when the primary key is accessed when it is found to be invalidated, then delete it
2. Active method (active way), periodically select a part of the failed primary key from the primary key that set the expiration time to delete
Internal representation of failure
Next, we will explore the implementation of these two methods through the code, but before we look at how the Redis management and maintenance of the primary key (note: The source code in this blog is all from Redis-2.6.12).
"Code snippet One" gives a definition of the structure of the database in Redis, which, in addition to the ID, is a pointer to a dictionary, where we look only at dict and expires, which maintain all key-value pairs contained in a REDIS database (the structure can be understood Is Dict[key]:value, that is, the mapping between the primary key and the value, and the latter is used to maintain a primary key (whose structure can be interpreted as expires[key]:timeout, a mapping of the primary key and the expiration time) that is set in the Redis database. When we insert data into the system using the Setex and Psetex commands, Redis first adds the key and Value to the dictionary table dict, and then adds the key and expiration time to the dictionary table of expires. When we use the EXPIRE, Expireat, Pexpire, and Pexpireat commands to set the expiration time of a primary key, Redis first go to dict this dictionary table to find out whether the primary key to be set exists, and if so, add the primary key and the expiration time to the expires This dictionary table. Simply put, the primary key and the specific expiration time that set the expiration time are all maintained in the Expires dictionary table.
"Code Snippet One":
Having a rough idea of how redis is maintaining a primary key that has been set to fail time, let's take a look at how Redis is implementing a negative deletion of the failed primary key. "Code snippet Two" gives a function called expireifneeded, which is invoked in any function that accesses data, meaning that Redis is invoked when all commands involving reading data are implemented, such as GET, Mget, Hget, Lrange, and so on. The significance of its existence is to read the data before it has no expiration, if it fails to delete it. "Code snippet Two" gives all the relevant descriptions of the expireifneeded function, and it doesn't repeat its implementation here. Here is another function called in the expireifneeded function Propagateexpire, which is used to broadcast information that the primary key has failed before formally deleting the failed primary key, which is propagated to two destinations: one is sent to the AoF file, The operation to delete the failed primary key is recorded in the standard command format of Del Key, and the other is to send all the Slave to the current Redis server, which will also remove the invalid primary key from the operation in the standard command format of Del Key to notify these Slave to delete their respective failed primary keys. From this we can know that all the Redis servers that run as Slave do not need to use negative methods to remove the failed primary key, they only need to master to be obedient to OK!
"Code snippet Two":
"Code snippet three":
The above is an introduction to the expireifneeded function to see how Redis removes the failed primary key in a negative way, but it is clearly not enough in this way, because if some of the failed primary keys are delayed to be accessed again, Redis You will never know that these primary keys have been invalidated and will never delete them, which would undoubtedly lead to a waste of memory space. As a result, Redis also prepares a positive deletion method, which uses Redis time events to implement a number of specified operations that are interrupted at intervals, including checking and deleting the failed primary key. Here we say the callback function for the time event is Servercron, which is created at Redis Server startup, and the number of executions per second is specified by the macro definition redis_default_hz, which is executed 10 times per second. Code snippet four gives the code for the program at the time the event was created, in the Initserver function of the redis.c file. In fact, Servercron this callback function not only for the failure of the primary key to check and delete, but also to update the statistics, client Connection timeout control, Bgsave and AOF trigger, etc., here we focus only on the deletion of the failure of the primary key implementation, that is, the function Activeexpirecycle.
"Code snippet Four":
"Code snippet Five":
Third, Memcached delete the failure of the primary key method and Redis what is the similarities and differences?
First, Memcached is also a negative way to remove a failed primary key, that is, the Memcached does not monitor whether the primary key is invalidated, but only if the primary key is accessed through a get. Second, the biggest difference between Memcached and Redis in the primary key failure mechanism is that Memcached does not really remove the failed primary key as Redis does, but simply reclaims the space occupied by the failed primary key. Thus, when new data is written to the system, Memcached takes precedence over the space of those failed primary keys. If the space for the failed primary key is used up, Memcached can also reclaim the space that has not been accessed for a long time through the LRU mechanism, so Memcached does not need a periodic delete operation like Redis, which is also determined by the memory management mechanism used by Memcached. At the same time, it should be pointed out that redis in the presence of Oom can also be configured maxmemory-policy This parameter to determine whether to use the LRU mechanism to reclaim the memory space (thanks to @jonathan_dai classmate in the Redis LRU mechanism of the original text of correction). In Redis, LRU is the default mechanism, and you might ask, if all keys are not set to expire, and Redis memory footprint is maxmemory, what happens when you add or modify keys? If there is no suitable key to remove, Redis will return an error when it is written. See Redis configuration files based on version 2.8
Redis failure mechanism of primary key will affect system performance?
Through the introduction of the Redis primary key failure mechanism, we know that although Redis will periodically check the primary key that set the expiration time and delete the failed primary key, but by limiting the number of databases per processing, the Activeexpirecycle function in one second, The limitation of the CPU time allocated to the Activeexpirecycle function, the limit of the number of failed primary keys that continues to be deleted, Redis has greatly reduced the effect of the primary key failure mechanism on the overall performance of the system, However, if there is a large number of primary keys in the actual application in a short period of time will also reduce the ability to respond to the system, so this situation should undoubtedly be avoided.
Start building with 50+ products and up to 12 months usage for Elastic Compute Service