Redis persistence policy Rdb and AOF

Redis Persistence:

Redis offers many different levels of persistence: one is an RDB and the other is aof.

RDB persistence can generate a point-in-time snapshot of a dataset (Point-in-time snapshot) within a specified time interval.

The AOF persists all the write commands that the server performs and restores the dataset by re-executing the commands when the server starts. The commands in the AOF file are all saved in the Redis protocol format, and the new command is appended to the end of the file. Redis can also override the AOF file in the background (rewrite) so that the volume of the AOF file does not exceed the actual size required to save the dataset State. Redis can also use both AOF persistence and RDB persistence. In this case, when the Redis restarts, it takes precedence over the AOF file to restore the dataset because the AOF file saves a dataset that is typically more complete than the data set saved by the RDB file. You can even turn off the persistence feature so that the data exists only when the server is running.

It is important to understand the similarities and differences between the RDB persistence and AOF persistence, which are described in detail in the following subsections, and are described in the same and different aspects of these two persistence features.

Advantages of the RDB:

An RDB is a very compact file that holds the data set of a Redis at a point in time. This file is ideal for backup: For example, you can back up an RDB file every hour within the last 24 hours, and also back up an RDB file every day of the month. In this case, you can restore the dataset to a different version at any time, even if you encounter problems. The RDB is ideal for disaster recovery (disaster recovery): It has only one file, and the content is very compact and can be transferred (after encryption) to another datacenter, or Amazon S3. An RDB maximizes the performance of Redis: The only thing the parent process has to do when saving an Rdb file is to fork out a child process, and the child process will handle all subsequent save work, and the parent process does not have to perform any disk I/O operations. An RDB recovers a large data set faster than AOF.

Disadvantages of the RDB:

If you need to avoid losing data when the server fails, the RDB is not for you. Although Redis allows you to set different savepoint (save point) to control how often an RDB file is saved, it is not an easy operation because the Rdb file needs to preserve the state of the entire dataset. Therefore, you may be able to save the RDB file at least 5 minutes. In this case, you may lose several minutes of data in the event of a failure. Each time you save an RDB, Redis will fork () out a subprocess and perform the actual persistence work by the child process. When the dataset is large, fork () can be time consuming, causing the server to stop processing the client in such milliseconds, and if the data set is huge and CPU time is very tight, the stop time may even be a full second. Although the AOF rewrite also requires fork (), there is no loss of data durability regardless of the length of the AOF rewrite execution interval.

Advantages of AOF:

For the same dataset, the volume of the AOF file is usually larger than the size of the RDB file. Depending on the Fsync policy you are using, the AOF may be slower than the RDB. In general, the performance of Fsync per second is still very high, while closing fsync can make AOF as fast as an RDB, even under high load. However, the RDB can provide a more guaranteed maximum delay time (latency) when handling large write loads. AOF has been a bug in the past: Because of the reason of individual commands, the AOF file cannot be restored as it was when the data set was reloaded. (for example, a blocking command Brpoplpush has caused such a bug.) The test suite adds tests to this situation: they automatically generate random, complex datasets, and re-load the data to make sure everything is fine. Although this bug is not common in AOF files, it is almost impossible for an RDB to appear in comparison.

RDB and AOF, which one should I use?

In general, if you want to achieve data security that is comparable to PostgreSQL, you should use two persistence features at the same time. If you are very concerned about your data, but can still tolerate data loss within a few minutes, you can use only the RDB persistence. Many users use AOF persistence only, but we do not recommend this approach: because the scheduled generation of an Rdb snapshot (snapshot) is very convenient for database backups, and the RDB recovers the data set faster than the AOF recovery, the use of an RDB also avoids the previously mentioned The bug of the AOF program. Because of the various reasons mentioned above, we may integrate AOF and RDB into a single persistence model in the future. (This is a long-term plan.) ）

RDB Snapshot:

By default, Redis saves a database snapshot in a binary file named Dump.rdb. You can set up Redis to automatically save a dataset once the condition that the dataset has at least M changes in N seconds is met. You can also manually let Redis do the dataset save operation by calling Save or BGSAVE. For example, the following settings will allow Redis to automatically save a dataset when it satisfies the condition "at least 1000 keys are changed in 60 seconds":
Save 60 1000
This persistence is called snapshot (snapshot).

How the Snapshot works:

When Redis needs to save the Dump.rdb file, the server performs the following actions:
Redis calls fork (), with both parent and child processes.
The child process writes the dataset to a temporary RDB file.
When a child process finishes writing to the new Rdb file, Redis replaces the original Rdb file with the new Rdb file and deletes the old Rdb file.
This way of working enables Redis to benefit from the write-time replication (copy-on-write) mechanism.
File only for append operation (append-only file,aof)
The snapshot feature is not very durable (durable): If Redis causes downtime for some reason, the server loses the data that was recently written and is still not saved to the snapshot. While the durability of data is not the most important consideration for some programs, the snapshot feature is less applicable for programs that pursue full durability.
Starting with version 1.1, Redis adds a completely durable way to persist: AOF persistence.
You can open the AOF feature by modifying the configuration file:
AppendOnly Yes
From now on, whenever Redis executes a command that changes the dataset (such as set), the command is appended to the end of the AOF file.
That way, when Redis restarts, the program can rebuild the dataset by re-executing the commands in the AOF file.

AOF rewrite:

Because AOF works by constantly appending commands to the end of a file, the volume of the AOF file becomes larger as the write command grows. For example, if you call a counter 100 times INCR, then just to save the current value of the counter, AOF file will need to use 100 records (entry). In practice, however, using only one SET command is sufficient to hold the current value of the counter, and the remaining 99 records are actually superfluous. To handle this situation, Redis supports an interesting feature: You can rebuild the AOF file without interrupting the service client (rebuild). To execute the bgrewriteaof command, Redis will generate a new AOF file that contains the fewest commands required to rebuild the current dataset.

How durable is AOF?
You can configure how long Redis will fsync data to disk once.
There are three options:
Each time a new command is appended to the AOF file, the Fsync is executed once: very slow and very secure.
Fsync per second: fast enough (almost as long as the RDB is persisted), and only 1 seconds of data are lost in the case of a failure.
Never Fsync: Give the data to the operating system for processing. Faster and less secure choice.
The recommended (and also the default) measure is Fsync per second, and this fsync strategy can take into account speed and security.
The strategy of always fsync is very slow in practice, even after Redis 2.0 has made improvements to the related programs-frequent calls to Fsync are destined to make this strategy impossible to get up to.

What if something goes wrong with the AOF file?

The server may be down when the program is writing to the AOF file, and if the outage causes a AOF file error (corrupt), then Redis will refuse to load the AOF file when it restarts, ensuring that data consistency is not compromised.

When this happens, you can fix the AOF file with the error in the following ways:

Create a backup of the existing AOF file.
Use the REDIS-CHECK-AOF program that came with Redis to fix the original aof file.
$ redis-check-aof–fix
Optionally, use Diff-u to compare the backup of the repaired AOF file and the original AOF file to see the differences between the two files.
Restart the Redis server, wait for the server to load the repaired AOF file, and perform data recovery.
How the AOF works
The AOF rewrite, like the RDB snapshot, cleverly leverages the write-time replication mechanism.

The following are the steps to perform the AOF rewrite:

Redis executes fork () and now has both parent and child processes.
The child process begins writing the contents of the new AOF file to a temporary file. For all newly executed write commands, the parent process accumulates them into a memory cache while appending these changes to the end of the existing AOF file: This allows the existing AOF files to be safe even if there is an outage in the middle of the rewrite. When a child process finishes rewriting work, it sends a signal to the parent process that the parent process appends all the data in the memory cache to the end of the new AOF file after it receives the signal. Now Redis atomically replaces the old file with the new file, and all commands are appended directly to the end of the new AOF file.

Create a backup for the latest Dump.rdb file.
Put the backup in a safe place.
Execute the following two commands:
redis-cli> CONFIG SET appendonly Yes
redis-cli> CONFIG SET Save ""
Ensure that the number of keys for the database has not changed since the command was executed.
Make sure that the Write command is appended to the end of the AOF file correctly.
The first command executed in step 3 turns on the AOF feature: Redis blocks until the initial AOF file creation is complete, and Redis continues to process the command request and begins appending the write command to the end of the AOF file.
Step 3 Executes the second command to turn off the RDB feature. This step is optional, and you can use both the RDB and AOF persistence features if you want.
Don't forget to open the AOF feature in redis.conf! Otherwise, after the server restarts, the configuration previously set by Config set will be forgotten, and the program will start the server as it was originally configured.

Interaction between the RDB and the AOF:

In Redis with a version number greater than or equal to 2.4, the BGSAVE does not perform bgrewriteaof during execution. Conversely, in the course of bgrewriteaof execution, BGSAVE cannot be executed.
This prevents two Redis background processes from doing large amounts of I/O to the disk at the same time.
If BGSAVE is executing and the user invokes the BGREWRITEAOF command, the server will reply to the user an OK state and inform the user that Bgrewriteaof has been scheduled to execute: Once the BGSAVE is executed, bgrewriteaof will be formally started. When Redis is started, if both RDB persistence and AOF persistence are turned on, the program takes precedence over the AOF file to recover the dataset, because the data that is stored in the AOF file is usually the most complete.

To back up Redis data:

Redis is very friendly for data backup because you can copy an RDB file when the server is running: Once the Rdb file is created, no modifications are made. When the server is about to create a new Rdb file, it saves the contents of the file in a temporary file, and when the temporary file is written, the program uses atomic to replace the original Rdb file with a temporary file. This means that, whenever an RDB file is copied, it is absolutely safe.

Transferred from: http://my.oschina.net/davehe/blog/174662

Redis persistence policy Rdb and AOF