Deep analysis of Redis persistence

Redis is a distributed NoSQL database system oriented to "Key-value" type data, which has the advantages of high performance, persistent storage, and adapting to the highly concurrent application scenarios. Although it started late, but the development is very rapid.
In recent days, Redis's author wrote in his blog, he saw all the discussion of Redis, the misunderstanding of Redis persistence is the biggest, so he wrote a long article to the redis of the persistence of a systematic discussion. The article contains three main aspects: how Redis persistence works, whether this performance is reliable, and compared with other types of databases. The following is the article content:

first, how Redis persistence is how to work.

What is persistence. Simply put the data in a device where data is not lost after a power outage, which is the hard drive that we normally understand. First, let's take a look at what the database does when it comes to writing, with the following five main processes:

The client sends a write operation to the server (the data is in the client's memory). The database server receives the data for the write request (the data is in the server's memory). The server calls the write system call to write the data to disk (the data is in the system memory buffer). The operating system transfers the data in the buffer to the disk controller (the data is in the disk cache). The disk controller writes data to the physical media on the disk (the data actually falls on disk). Fault analysis

Write operation is roughly 5 processes above, we combine the above 5 processes to look at various levels of failure:

When the database system fails, the system kernel is still intact. So as soon as we finish step 3rd, the data is secure because the subsequent operating system will complete the next few steps to ensure that the data will eventually fall on disk. When the system is powered down, all the caches mentioned in the top 5 are invalidated, and the database and operating system will stop working.so only when the data is completed in the 5th step can we guarantee that the data will not be lost after power outage。 With the 5 steps above, perhaps we would like to find out some of the following questions:

How often the database calls write and writes the data to the kernel buffer. How long the kernel writes data from the system buffer to the disk controller. When the disk controller writes the data from the cache to the physical media. For the first question, the database level is usually fully controlled. For the second problem, the operating system has its default policy, but we can also force the operating system to write data from the kernel to disk controllers through the Fsync series of commands provided by the POSIX API. For the third problem, it seems as if the database is out of reach, but in fact, most of the time the disk cache is set to shut down, or it is only open for read caching, which means that the write operation is not cached and written directly to the disk. The recommended approach is to open the write cache only if your disk device has a standby battery.

Data corruption

The so-called data corruption, is the data can not be recovered, above we are talking about how to ensure that the data is actually written to disk, but writing to disk may not mean that the data will not be damaged. For example, we may write a request two times a different write operation, when the accident, may cause a write operation safely completed, but another time has not been done. If the data file structure of the database is not organized properly, it may result in a situation where data is completely unrecoverable.

There are usually three strategies for organizing data to prevent data files from being corrupted to unrecoverable conditions:

The first is the most coarse processing, is not through the organization of data to ensure the recoverability of data. Instead, a data backup is used to restore the data file after it has been corrupted by configuring the data to synchronize the backup. In fact MongoDB does not open the operation log, this is the case by configuring replica sets. The other is to add an action log based on the above, to remember the behavior of the operation each time, so that we can use the action log for data recovery. Because the action log is written sequentially, there is no case where the action log cannot be recovered. This is similar to the case where the MongoDB opened the operation log. More insurance is that the database does not make old data changes, just append to complete the write operation, so that the data itself is a log, so that the data can never be restored. In fact, COUCHDB is an excellent example of this practice. 1. Redis's first persistence strategy: RDB Snapshots

Redis supports the persistence mechanism of saving snapshots of the current data into a data file. And how does a persistent write database generate a snapshot? Redis with the copy on write mechanism of the fork command. When the snapshot is generated, fork the current process out of a subprocess, and then loops through all the data in the subprocess, writing the data to the Rdb file.

We can configure the timing of the RDB snapshot generation with the Redis save instruction, such as you can configure the snapshot to be generated 100 times within 10 minutes, or you can configure a snapshot to be generated 1000 times in 1 hours, or multiple rules to implement together. These rules are defined in the Redis configuration file, and you can also set the rules at Redis run time using the Redis config set command without restarting the Redis.

Redis's Rdb file won't break, because the write operation is done in a new process, when a new Rdb file is generated, the Redis-generated subprocess writes the data to a temporary file and then renames the temporary file to the Rdb file through an atomic rename system call, In this way, Redis RDB files are always available when a failure occurs at any time.

At the same time, Redis's Rdb file is also a link in the internal implementation of Redis master-slave synchronization.

But, we can obviously see, RDB has its insufficiency, is once the database problem, then our RDB file saves the data is not brand-new, from the last Rdb file generation to Redis downtime this period of time data all lost. This can be tolerated in some businesses, and we recommend that these businesses use RDB to persist because the cost of opening RDB is not high. But for other applications that require very high data security and cannot tolerate the application of data loss, RDB can do nothing, so Redis introduces another important persistence mechanism: AOF log.

2. Redis's second persistence strategy: AOF Log

AoF Log Full name is append only file, from the name we can see, it is an append write log file. Unlike a general database, the AoF file is an identifiable plain text, and its content is a Redis standard command. For example, we do the following experiments, using the Redis2.6 version, in the startup command parameters set to open the AOF function:

Code./redis-server--appendonly Yes
Then we execute the following command:

Code Redis 127.0.0.1:6379> Set key1 Hello OK redis 127.0.0.1:6379> append key1 "world!"  (integer) Redis 127.0.0.1:6379> del key1 (integer) 1 redis 127.0.0.1:6379> del non_existing_key (integer) 0
When we look at the aof log file, we get the following:

Code $ cat appendonly.aof *2 $ $0 *3 $ set $ Key1 $ Hello *3 $ append $ ke   Y1 $ world! *2 $ del $ key1
As you can see, the write operation generates a corresponding command as a log. It is noteworthy that the last del command, which is not recorded in the AoF log, is because Redis determines that the command does not make changes to the current dataset. So there's no need to record this useless write command. In addition, the AOF log is not fully generated by the client's request. For example, the command incrbyfloat is recorded as a set record when the AoF log is recorded, because the floating-point operation may be different on different systems, so in order to prevent the same log from generating different datasets on different systems, So here only the results of the operation are recorded through the set.

AoF rewrite

You can think that each write command generates a log, so the aof file is not very large. The answer is yes, the AoF file will grow larger and bigger,