Redis Persistent RDB and aof-reprint

Redis Persistence:

Provides several different levels of persistence: one is an RDB and the other is aof.

RDB persistence can generate a point-in-time snapshot of a dataset (Point-in-time snapshot) within a specified time interval.

The AOF persists all the write commands that the server performs and restores the dataset by re-executing the commands when the server starts. The commands in the AOF file are all saved in the Redis protocol format, and the new command is appended to the end of the file. Redis can also override the AOF file in the background (rewrite) so that the volume of the AOF file does not exceed the actual size required to save the dataset State. Redis can also use both AOF persistence and RDB persistence. In this case, when the Redis restarts, it takes precedence over the AOF file to restore the dataset because the AOF file saves a dataset that is typically more complete than the data set saved by the RDB file. You can even turn off the persistence feature so that the data exists only when the server is running.

It is important to understand the similarities and differences between the RDB persistence and AOF persistence, which are described in detail in the following subsections, and are described in the same and different aspects of these two persistence features.

Advantages of the RDB:
An RDB is a very compact file that holds the data set of a Redis at a point in time. This file is ideal for backup: For example, you can back up an RDB file every hour within the last 24 hours, and also back up an RDB file every day of the month. In this case, you can restore the dataset to a different version at any time, even if you encounter problems. The RDB is ideal for disaster recovery (disaster recovery): It has only one file, and the content is very compact and can be transferred (after encryption) to another datacenter, or Amazon S3. An RDB maximizes the performance of Redis: The only thing the parent process has to do when saving an Rdb file is to fork out a child process, and the child process will handle all subsequent save work, and the parent process does not have to perform any disk I/O operations. An RDB recovers a large data set faster than AOF.

The disadvantage of the RDB:
If you need to avoid losing data when the server fails, the RDB is not for you. Although Redis allows you to set different savepoint (save point) to control how often an RDB file is saved, it is not an easy operation because the Rdb file needs to preserve the state of the entire dataset. Therefore, you may be able to save the RDB file at least 5 minutes. In this case, you may lose several minutes of data in the event of a failure. Each time you save an RDB, Redis will fork () out a subprocess and perform the actual persistence work by the child process. When the dataset is large, fork () can be time consuming, causing the server to stop processing the client in such milliseconds, and if the data set is huge and CPU time is very tight, the stop time may even be a full second. Although the AOF rewrite also requires fork (), there is no loss of data durability regardless of the length of the AOF rewrite execution interval.

AOF Benefits:
Using AOF persistence makes Redis very durable (much more durable): You can set different fsync policies, such as no Fsync, Fsync per second, or fsync each time a write command is executed. The default policy for AOF is Fsync once per second, in which Redis can still maintain good performance and, even if there is a failure, it will lose at most one second of data (Fsync will execute in a background thread, so the main thread can continue to work on the command request). The AOF file is an append-only log file (append only), so the write to the AOF file does not require seek, even if the log contains incomplete commands for some reason (such as a write-in disk full, write-down, etc.), The Redis-check-aof tool can also easily fix this problem.
Redis can automatically override AOF in the background when the AOF file volume becomes too large: The rewritten new AOF file contains the minimum set of commands required to restore the current dataset. The entire rewrite operation is absolutely secure, as Redis will continue to append commands to the existing AOF file during the creation of the new AOF file, and the existing AOF files will not be lost even if there is an outage during the rewrite. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and begins appending to the new AOF file. The AOF file preserves all write operations performed on the database in an orderly manner, and these writes are saved in the format of the Redis protocol, so the contents of the AOF file are easily readable and parsing (parse) is easy. The export AOF file is also very simple: for example, if you accidentally executed the Flushall command, but as long as the AOF file is not rewritten, just stop the server, remove the Flushall command at the end of the AOF file, and restart Redis, you can The data set reverts to the state before Flushall execution.

Disadvantages of AOF:
For the same dataset, the volume of the AOF file is usually larger than the size of the RDB file. Depending on the Fsync policy you are using, the AOF may be slower than the RDB. In general, the performance of Fsync per second is still very high, while closing fsync can make AOF as fast as an RDB, even under high load. However, the RDB can provide a more guaranteed maximum delay time (latency) when handling large write loads. AOF has been a bug in the past: Because of the reason of individual commands, the AOF file cannot be restored as it was when the data set was reloaded. (for example, a blocking command Brpoplpush has caused such a bug.) The test suite adds tests to this situation: they automatically generate random, complex datasets, and re-load the data to make sure everything is fine. Although this bug is not common in AOF files, it is almost impossible for an RDB to appear in comparison.

RDB and AOF, which one should I use?
In general, if you want to achieve data security that is comparable to PostgreSQL, you should use two persistence features at the same time. If you are very concerned about your data, but can still tolerate data loss within a few minutes, you can use only the RDB persistence. Many users use AOF persistence only, but we do not recommend this approach: because the scheduled generation of an Rdb snapshot (snapshot) is very convenient for database backups, and the RDB recovers the data set faster than the AOF recovery, the use of an RDB also avoids the previously mentioned The bug of the AOF program. Because of the various reasons mentioned above, we may integrate AOF and RDB into a single persistence model in the future. (This is a long-term plan.) ）

RDB Snapshot:
By default, Redis saves a database snapshot in a binary file named Dump.rdb. You can set up Redis to automatically save a dataset once the condition that the dataset has at least M changes in N seconds is met. You can also manually let Redis do the dataset save operation by calling Save or BGSAVE. For example, the following settings will allow Redis to automatically save a dataset when it satisfies the condition "at least 1000 keys are changed in 60 Seconds":
Save
is called a snapshot (snapshot).
How snapshots work:
When Redis needs to save the Dump.rdb file, the server does the following:
Redis calls fork (), and both parent and child processes. The
Child process writes the dataset to a temporary RDB file.
When a child process finishes writing to the new Rdb file, Redis replaces the original Rdb file with the new Rdb file and deletes the old Rdb file.
This way of working enables Redis to benefit from the write-time replication (copy-on-write) mechanism. The
Append-only file (append-only file,aof)
Snapshot feature is not very durable (durable): If Redis causes downtime for some reason, the server loses the data that was most recently written and still not saved to the snapshot. While the durability of data is not the most important consideration for some programs, the snapshot feature is less applicable for programs that pursue full durability.
Starting with version 1.1, Redis adds a completely durable way to persist: AOF persistence.
You can modify the configuration file to open the AOF feature:
appendonly Yes
from now on, whenever Redis executes a command that changes the dataset (such as SET), the command is appended to the end of the AOF file.
this way, when Redis restarts, the program can achieve the purpose of rebuilding the dataset by re-executing the commands in the AOF file.

AOF rewrite:
Because AOF works by constantly appending commands to the end of a file, the volume of the AOF file becomes larger as the write command grows. For example, if you call a counter 100 times INCR, then just to save the current value of the counter, AOF file will need to use 100 records (entry). In practice, however, using only one SET command is sufficient to hold the current value of the counter, and the remaining 99 records are actually superfluous. To handle this situation, Redis supports an interesting feature: You can rebuild the AOF file without interrupting the service client (rebuild). To execute the bgrewriteaof command, Redis will generate a new AOF file that contains the fewest commands required to rebuild the current dataset.

How durable is AOF?
You can configure how long Redis will fsync data to disk once.
There are three options:
Each time a new command is appended to the AOF file, the Fsync is executed once: very slow and very secure.
Fsync per second: fast enough (almost as long as the RDB is persisted), and only 1 seconds of data are lost in the case of a failure.
Never Fsync: Give the data to the operating system for processing. Faster and less secure choice.
The recommended (and also the default) measure is Fsync per second, and this fsync strategy can take into account speed and security.
The strategy of always fsync is very slow in practice, even after Redis 2.0 has made improvements to the related programs-frequent calls to Fsync are destined to make this strategy impossible to get up to.

What if the AOF file is wrong? The
server may be down while the program is writing to the AOF file, and if the outage causes a AOF file error (corrupt), Redis will refuse to load the AOF file when it restarts, ensuring that data consistency is not compromised.
When this occurs, you can use the following methods to fix the AOF file that was faulted:
Create a backup of the existing AOF file.
Use the REDIS-CHECK-AOF program that came with Redis to fix the original aof file.
$ redis-check-aof--fix
(optional) Use Diff-u to compare the backup of the repaired AoF file and the original aof file to see the differences between the two files.
Restart the Redis server, wait for the server to load the repaired AOF file, and perform data recovery.
How AOF works
AOF rewrite and RDB create snapshots, all subtly exploit the write-time replication mechanism.

The following steps are performed by the AOF override:
Redis executes fork () and now has both parent and child processes. The
child process begins writing the contents of the new AOF file to a temporary file. For all newly executed write commands, the parent process accumulates them into a memory cache while appending these changes to the end of the existing AOF file: This allows the existing AOF files to be safe even if there is an outage in the middle of the rewrite. When a child process finishes rewriting work, it sends a signal to the parent process that the parent process appends all the data in the memory cache to the end of the new AOF file after it receives the signal. Now Redis atomically replaces the old file with the new file, and all commands are appended directly to the end of the new AOF file. The
creates a backup for the most recent Dump.rdb file. The
puts the backup in a safe place. The
executes the following two commands:
redis-cli> config set appendonly yes
redis-cli> config set save ""
ensure that the number of keys for the database has not changed since the command was executed. The
ensures that the Write command is correctly appended to the end of the AOF file.
The first command executed in step 3 turns on the AOF feature: Redis blocks until the initial AOF file creation is complete, and Redis continues to process the command request and begins appending the write command to the end of the AOF file. The second command that
Step 3 executes is used to turn off the RDB feature. This step is optional, and you can use both the RDB and AOF persistence features if you want.
Don't forget to open the AOF feature in redis.conf! Otherwise, after the server restarts, the configuration previously set by Config set will be forgotten, and the program will start the server as it was originally configured.

Interaction between the RDB and the AOF:
In Redis with a version number greater than or equal to 2.4, the BGSAVE does not perform bgrewriteaof during execution. Conversely, in the course of bgrewriteaof execution, BGSAVE cannot be executed.
This prevents two Redis background processes from doing large amounts of I/O to the disk at the same time.
If BGSAVE is executing and the user invokes the BGREWRITEAOF command, the server will reply to the user an OK state and inform the user that Bgrewriteaof has been scheduled to execute: Once the BGSAVE is executed, bgrewriteaof will be formally started. When Redis is started, if both RDB persistence and AOF persistence are turned on, the program takes precedence over the AOF file to recover the dataset, because the data that is stored in the AOF file is usually the most complete.

To back up Redis data:
Redis is very friendly for data backup because you can copy an RDB file when the server is running: Once the Rdb file is created, no modifications are made. When the server is about to create a new Rdb file, it saves the contents of the file in a temporary file, and when the temporary file is written, the program uses atomic to replace the original Rdb file with a temporary file. This means that, whenever an RDB file is copied, it is absolutely safe.

Redis Persistent RDB and aof-reprint