NoSQL Redis Persistence

If you think that Redis is a key value store, it might be used instead of MySQL, and if you think of it as a persistent cache, it might just save some of the temporary data that is frequently accessed.

Let's think of a question that, by the previous introduction, we know that Redis and memcached can be used as caches to improve access efficiency, so if there is a sudden power outage or other failure, does that mean our cache data is lost?

Redis is an in-memory database that supports persistence, which means that Redis often needs to synchronize in-memory data to disk to ensure persistence. That means that the cache interacts with our database, and we can save the data in the cache, which is where Redis is more powerful than other caches: persistence.

Redis supports two persistence modes, one of which is the RDB (snapshot) and the default, and the other is the way Append-only file (abbreviated AOF).
RDB persistence can generate a point-in-time snapshot of a dataset (Point-in-time snapshot) within a specified time interval.

The AOF persists all the write commands that the server performs and restores the dataset by re-executing the commands when the server starts. The commands in the AOF file are all saved in the Redis protocol format, and the new command is appended to the end of the file. Redis can also override the AOF file in the background (rewrite) so that the volume of the AOF file does not exceed the actual size required to save the dataset State.

Redis can also use both AOF persistence and RDB persistence. In this case, when the Redis restarts, it takes precedence over the AOF file to restore the dataset because the AOF file saves a dataset that is typically more complete than the data set saved by the RDB file. You can even turn off the persistence feature so that the data exists only when the server is running.

Rdb

Snapshots are the default persistence mode. This is the way in which the in-memory data is written to the binary in a snapshot, the default file name is Dump.rdb. You can automatically make snapshot persistence by configuring settings.
We can configure Redis to take snapshots automatically if more than M key is modified in n seconds, the following is the default snapshot save configuration

9001  #900秒内如果超过1个key被修改，则发起快照保存30010#300秒内容如超过10个key被修改，则发起快照保存6010000

RDB Snapshot Operation Process

By default, Redis saves a database snapshot in a binary file named Dump.rdb. You can set up Redis to automatically save a dataset once the condition that the dataset has at least M changes in N seconds is met.

You can also manually let Redis do the dataset save operation by calling Save or BGSAVE. For example, the following settings will allow Redis to automatically save a dataset when it satisfies the condition "at least 1000 keys are changed in 60 seconds":

save 60 1000

When Redis needs to save the Dump.rdb file, the server performs the following actions:

Redis calls fork (), with both parent and child processes.

The child process writes the dataset to a temporary RDB file.
When a child process finishes writing to the new Rdb file, Redis replaces the original Rdb file with the new Rdb file and deletes the old Rdb file.

This way of working enables Redis to benefit from the write-time replication (copy-on-write) mechanism.

File only for append operation (append-only file,aof)

AOF

Starting with version 1.1, Redis adds a completely durable way to persist: AOF persistence.
You can open the AOF feature by modifying the configuration file:

yes

From now on, whenever Redis executes a command that changes the dataset (such as set), the command is appended to the end of the AOF file.

That way, when Redis restarts, the program can rebuild the dataset by re-executing the commands in the AOF file.

It is also important to note that each snapshot persistence is a full write of the memory data to disk once, not the incremental synchronization of only dirty data. If the amount of data is large, and more write operations, it will inevitably cause a large number of disk IO operations, may seriously affect performance.

In addition, because the snapshot is done at a certain interval, if Redis is accidentally down, all changes after the last snapshot will be lost. You can use AOF persistence if your application requires that you cannot lose any modifications.

AOF Operation Process

AOF is more persistent than snapshot mode, because Redis appends each received write command to a file by using the Write function (default is appendonly.aof) when aof persistence is used.

When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands saved in the file. Of course, because the OS caches write modifications in the kernel, it may not be written to disk immediately. The persistence of this aof method is also likely to lose some of the modifications.

But we can tell Redis through the configuration file that we want to force the OS to write to disk through the Fsync function. There are three ways to do this (the default is: Fsync once per second)

# appendfsync always      //每次收到写命令就立即强制写入磁盘，最慢的，但是保证完全的持久化，不推荐使用

appendfsync everysec     //每秒钟强制写入磁盘一次，在性能和持久化方面做了很好的折中，推荐

# appendfsync no    //完全依赖os，性能最好,持久化没保证

The specific process is as follows
1. Redis Call Fork, now has a parent-child two processes

2. The child process writes to the temporary file the command to rebuild the database state based on the database snapshot in memory

3. The parent process continues to process the client request, in addition to writing the write command to the original aof file. Cache the received write commands at the same time. This will ensure that if the child process rewrite fails, it will not be problematic.

4. When a child process writes the snapshot content to a temporary file, the child process signals the parent process. The parent process then writes the cached write command to the temporary file as well.

5. Now the parent process can replace the old aof file with the temporary file and rename it, and the subsequent write commands are also started to append to the new aof file.

Note that the operation to rewrite the aof file does not read the old aof file, but instead overwrites the entire in-memory database content with a new aof file in the form of a command, which is a bit similar to a snapshot.

After installing Redis, we'll find the redis.conf file, where we can set both of these persistence methods.

Advantages of an RDB

An RDB is a very compact file that holds the data set of a Redis at a point in time. This file is ideal for backup: For example, you can back up an RDB file every hour within the last 24 hours, and also back up an RDB file every day of the month.

In this case, you can restore the dataset to a different version at any time, even if you encounter problems. The RDB is ideal for disaster recovery (disaster recovery): It has only one file, and the content is very compact and can be transferred (after encryption) to another datacenter, or Amazon S3.

An RDB maximizes the performance of Redis: The only thing the parent process has to do when saving an Rdb file is to fork out a child process, and the child process will handle all subsequent save work, and the parent process does not have to perform any disk I/O operations. An RDB recovers a large data set faster than AOF.

Disadvantages of the RDB

If you need to avoid losing data when the server fails, the RDB is not for you. Although Redis allows you to set different savepoint (save point) to control how often an RDB file is saved, it is not an easy operation because the Rdb file needs to preserve the state of the entire dataset. Therefore, you may be able to save the RDB file at least 5 minutes.

In this case, you may lose several minutes of data in the event of a failure. Each time you save an RDB, Redis will fork () out a subprocess and perform the actual persistence work by the child process.

When the dataset is large, fork () can be time consuming, causing the server to stop processing the client in such milliseconds, and if the data set is huge and CPU time is very tight, the stop time may even be a full second. Although the AOF rewrite also requires fork (), there is no loss of data durability regardless of the length of the AOF rewrite execution interval.

Advantages of AOF

Using AOF persistence makes Redis very durable (much more durable): You can set different fsync policies, such as no fsync, one fsync per second, or Fsync each time you execute a write command.

The default policy for AOF is Fsync once per second, in which Redis can still maintain good performance and, even if there is a failure, it will lose at most one second of data (Fsync will execute in a background thread, so the main thread can continue to work on the command request).

The AOF file is an append-only log file (append only), so the write to the AOF file does not require seek, even if the log contains incomplete commands for some reason (such as a write-in disk full, write-down, etc.), The Redis-check-aof tool can also easily fix this problem.

Redis can automatically override AOF in the background when the AOF file size becomes too large: The rewritten new AOF file contains the minimum set of commands required to restore the current dataset. The entire rewrite operation is absolutely secure, as Redis will continue to append commands to the existing AOF file during the creation of the new AOF file, and the existing AOF files will not be lost even if there is an outage during the rewrite. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and begins appending to the new AOF file.

The AOF file preserves all write operations performed on the database in an orderly manner, and these writes are saved in the format of the Redis protocol, so the contents of the AOF file are easily readable and parsing (parse) is easy. The export AOF file is also very simple: for example, if you accidentally executed the Flushall command, but as long as the AOF file is not rewritten, just stop the server, remove the Flushall command at the end of the AOF file, and restart Redis, you can The data set reverts to the state before Flushall execution.

Disadvantages of AOF

For the same dataset, the volume of the AOF file is usually larger than the size of the RDB file. Depending on the Fsync policy you are using, the AOF may be slower than the RDB.

In general, the performance of Fsync per second is still very high, while closing fsync can make AOF as fast as an RDB, even under high load. However, the RDB can provide a more guaranteed maximum delay time (latency) when handling large write loads.

AOF has been a bug in the past: Because of the reason of individual commands, the AOF file cannot be restored as it was when the data set was reloaded. (for example, a blocking command Brpoplpush has caused such a bug.) The test suite adds tests to this situation: they automatically generate random, complex datasets, and re-load the data to make sure everything is fine. Although this bug is not common in AOF files, it is almost impossible for an RDB to appear in comparison.

RDB and AOF, which one should I use?

In general, if you want to achieve data security that is comparable to PostgreSQL, you should use two persistence features at the same time. If you are very concerned about your data, but can still tolerate data loss within a few minutes, you can use only the RDB persistence. Many users use AOF persistence only, but we do not recommend this approach: because the scheduled generation of an Rdb snapshot (snapshot) is very convenient for database backups, and the RDB recovers the data set faster than the AOF recovery, the use of an RDB also avoids the previously mentioned The bug of the AOF program.

NoSQL Redis Persistence