Redis persistence)

Redis persistence

Redis provides multiple levels of Persistence:

• RDB persistence allows you to generate a point-in-time snapshot of a dataset within a specified interval ).
• Aof records all write operation commands executed by the server persistently, and re-executes these commands to restore the dataset when the server starts. All commands in the aof file are saved in the format of the redis protocol. The new commands are appended to the end of the file. Redis can also rewrite the aof file in the background so that the size of the aof file does not exceed the actual size required to save the dataset status.
• Redis can also use both aof persistence and RDB persistence. In this case, when redis restarts, it will first use the aof file to restore the data set, because the data set saved in the aof file is generally more complete than the data set saved in the RDB file.
• You can even disable the persistence function so that data exists only when the server is running.

It is very important to understand the similarities and differences between RDB persistence and aof persistence. The following sections detail the two persistence functions and describe their similarities and differences.

Advantages of RDB

• RDB is a compact file that stores redis data sets at a certain point in time. This type of file is very suitable for backup: for example, you can back up the RDB file every hour within the last 24 hours, and back up an RDB file every day of each month. In this way, the dataset can be restored to different versions at any time even if a problem occurs.
• RDB is very suitable for disaster recovery (disaster recovery): it only has one file, and the content is very compact, it can be (encrypted) transmitted to another data center, or Amazon S3.
• RDB can maximize redis performance: the only thing the parent process has to do when saving RDB filesforkA sub-process is generated, and the sub-process will process all subsequent save tasks. The parent process does not need to perform any disk I/O operations.
• RDB is faster than aof in recovering big data sets.

Disadvantages of RDB

• If you need to avoid data loss when the server fails, RDB is not suitable for you. Although redis allows you to set different save points to control the frequency of saving RDB files, because the RDB file needs to save the status of the entire dataset, therefore, it is not an easy operation. Therefore, you may save the RDB file at least five minutes. In this case, you may lose data for several minutes in the event of a fault or shutdown.
• Every time you save the RDB, redis needsfork()Generates a sub-process and the sub-process performs the actual persistence work. When the dataset is large,fork()It may take a lot of time, causing the server to stop processing the client within a certain millisecond. If the dataset is huge and the CPU time is very short, the stop time may even be up to a whole second. Aof rewrite is also required.fork()But no matter how long the execution interval of aof rewrite is, there will be no loss in data durability.

Aof advantages

• Using aof persistence will make redis very durable (much more durable): You can set differentfsyncPolicy, such as nonefsync, Once every secondfsyncOr every time you execute a write commandfsync. The default policy of aof is per second.fsyncOnce, in this configuration, redis can still maintain good performance, and even if a fault is stopped, it will lose up to one second of data (fsyncWill be executed in the background thread, so the main thread can continue to process command requests ).
• The aof file is an append-only log file. Therefore, writing aof files is not required.seekEven if the log contains incomplete commands for some reason (for example, when the disk is full and the write is stopped midway through ),redis-check-aofThe tool can also easily fix this problem.
• Redis can automatically rewrite aof in the background when the aof file size is too large: The New aof file after rewriting contains the minimum command set required to restore the current dataset. The entire rewrite operation is absolutely safe, because redis will continue to append the command to the existing aof file during the creation of the aof file, even if there is a shutdown during the rewrite process, existing aof files will not be lost. Once the new aof file is created, redis will switch from the old aof file to the new aof file and start appending the new aof file.
• The aof file stores all write operations performed on the database in an orderly manner. These write operations are saved in the format of the redis protocol. Therefore, the content of the aof file is very easy to understand, it is also easy to analyze files (PARSE. The Export (export) aof file is also very simple: for example, if you accidentally execute the flushall command, but as long as the aof file is not overwritten, you only need to stop the server, remove the flushall command at the end of the aof file and restart redis to restore the data set to the State before the execution of flushall.

Disadvantages of aof

• For the same dataset, the size of the aof file is usually larger than that of the RDB file.
• According tofsyncPolicy, aof speed may be slower than RDB. In generalfsyncThe performance is still very high, and the function is disabledfsyncThe aof speed can be as fast as RDB, even under high load. However, when processing massive write loads, RDB can provide a more guaranteed maximum latency (latency ).
• Aof has encountered such a bug in the past: Due to some commands, when the aof file is re-loaded, it cannot restore the dataset to its original state when it is saved. (For example, the blocking command brpoplpush has caused such a bug .) Tests are added to the test suite for this situation: they automatically generate random and complex datasets and ensure that everything works by re-loading the data. Although this bug is not common in aof files, RDB is almost impossible to produce this bug.

RDB and aof, which one should I use?

In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence functions at the same time.

If you are very concerned about your data, but still can withstand data loss within several minutes, you can only use RDB for persistence.

Many users only use aof persistence, but we do not recommend this method: it is very easy to back up databases because regular generation of RDB snapshots (snapshot, in addition, RDB can recover data sets faster than Aof. In addition, RDB can avoid the previous aof program bugs.

 

RDB Snapshot

By default, redis saves the database snapshot indump.rdb.

You can set redis toNWithin seconds, the dataset must have at least oneMWhen this condition is met, the dataset is automatically saved.

You can also manually ask redis to save the dataset by calling save or bgsave.

For example, the following settings enable redis to60There are at least1000Key changed ", the dataset is automatically saved once:

save 60 1000

This persistence method is called snapshot ).

Snapshot Operation Method

When redis needs to savedump.rdbThe server performs the following operations:

1. Redis callfork()And has both parent and child processes.
2. The child process writes the dataset to a temporary RDB file.
3. When the sub-process completes writing to the new RDB file, redis replaces the original RDB file with the new RDB file and deletes the old RDB file.

This method allows redis to benefit from the copy-on-write mechanism.

Append-only file (AOF)

The Snapshot feature is not durable: If redis is down due to a fault for some reason, the server will lose the data that has been recently written and has not been saved to the snapshot.

Although the durability of data is not the most important consideration for some programs, the snapshot function is not suitable for those who are pursuing full durability.

Starting from Version 1.1, redis has added a completely durable persistence method: aof persistence.

You can modify the configuration file to enable the aof function:

appendonly yes

From now on, every time redis executes a command to change the dataset (such as set), this command will be appended to the end of the aof file.

In this way, when redis is restarted, the program can re-execute the commands in the aof file to recreate the dataset.

Aof rewrite

Because aof constantly appends commands to the end of a file, the size of the aof file increases as the number of write commands increases.

For example, if you call incr for a counter for 100 times, you only need to use 100 entries in the aof file to save the current value of the counter ).

However, in fact, only one set command is enough to save the current value of the counter, and the other 99 records are actually redundant.

To handle this situation, redis supports an interesting feature: you can rebuild the aof file without interrupting the service client ).

Run the bgrewriteaof command. redis will generate a new aof file, which contains the minimum command required to reconstruct the current dataset.

Redis 2.2 needs to manually execute the bgrewriteaof command; redis 2.4 can automatically trigger aof rewrite. For details, see the sample configuration file in redis 2.4.

What is the durability of aof?

How long can you configure redis to collect data?fsyncTo the disk once.

There are three options:

• Each time a new command is appended to the aof file, it is executed once.fsync: Very slow and secure.
• Per secondfsyncOnce: fast enough (similar to using RDB for persistence), and only one second of data will be lost in case of a fault.
• Neverfsync: Hand over the data to the operating system for processing. Faster and less secure options.

Recommended (and also default) measures per secondfsyncOnce, suchfsyncThe policy can take into account both speed and security.

AlwaysfsyncThe policy is very slow in actual use, even after redis 2.0 has improved the related program.fsyncIt is doomed that such a strategy cannot get faster.

What if an error occurs in the aof file?

The server may be shut down when the program is writing aof files. If the aof file is faulty, redis will refuse to load the aof file when it is restarted, this ensures that data consistency is not damaged.

In this case, you can use the following methods to fix the aof file:

1. Create a backup for the existing aof file.
2. Useredis-check-aofProgram to fix the original aof file.

$ redis-check-aof --fix

3. (Optional) Usediff -uCompare the backup of the repaired aof file and the original aof file to view the differences between the two files.
4. Restart the redis server, wait for the server to load the repaired aof file, and recover the data.

Aof Operation Method

Like creating snapshots in RDB, aof rewrite uses the write-time replication mechanism cleverly.

The following describes how to rewrite aof:

1. Redis executionfork()And now has both parent and child processes.
2. The child process starts to write the content of the new aof file to the temporary file.
3. For all newly executed write commands, the parent process accumulates them into a memory cache and appends these changes to the end of the existing aof file: in this way, the existing aof files are still safe even when the rewrite is interrupted.
4. When a child process completes rewriting, it sends a signal to the parent process. After receiving the signal, the parent process appends all data in the memory cache to the end of the new aof file.
5. Done! Now redis replaces the old file with the new file atomically, and all subsequent commands will be directly appended to the end of the new aof file.

How to switch from RDB persistence to aof persistence

In redis 2.2 or later versions, you can switch from RDB to aof without restarting:

1. Is the latestdump.rdbFile to create a backup.
2. Put the backup in a safe place.
3. Run the following two commands:

redis-cli> CONFIG SET appendonly yesredis-cli> CONFIG SET save ""

4. Make sure that the number of database keys does not change after the command is executed.
5. Make sure that the write command is correctly appended to the end of the aof file.

The aof function is enabled for the First Command executed in step 3: redis will block until the initial aof file creation is complete, and redis will continue to process the command request, and start to append the write command to the end of the aof file.

The second command executed in step 3 is used to disable the RDB function. This step is optional. You can also use both the RDB and aof persistence functions if you want.

Interaction between RDB and aof

In redis with a version number greater than or equal to 2.4, bgrewriteaof cannot be executed during bgsave execution. Likewise, bgsave cannot be executed during bgrewriteaof execution.

This prevents two redis background processes from simultaneously performing a large number of I/O operations on the disk.

If bgsave is being executed and the user explicitly calls the bgrewriteaof command, the server will reply to the userOKStatus, and inform the user that bgrewriteaof has been scheduled to be executed: Once bgsave is completed, bgrewriteaof will officially start.

When redis is started, if both RDB persistence and aof persistence are enabled, the program will first use the aof file to restore the dataset, because the data stored in the aof file is usually the most complete.

Back up redis data

Before reading this section, remember the following sentence: Be sure to back up your database!

Disk faults, node failures, and other such problems may make your data disappear, and it is very dangerous not to back up your data.

Redis is very friendly for data backup, because you can copy the RDB file when the server is running: Once the RDB file is created, it will not be modified. When the server creates a new RDB file, it first saves the file content in a temporary file. When the temporary file is written, the program usesrename(2)Replace the original RDB file with a temporary file atomically.

This means that, at any time, copying RDB files is absolutely safe.

Our suggestions are as follows:

• Create a scheduled job to back up an RDB file to a folder every hour and back up an RDB file to another folder every day.
• Make sure that the snapshot backup has the corresponding date and time information.findCommand to delete expired snapshots: for example, you can retain the snapshots of each hour in the last 48 hours, or the snapshots of each day in the last one or two months.
• Back up RDB to external data centers at least once a day, or at least to physical machines running your redis server.

Disaster Recovery Backup

Redis's Disaster Recovery Backup basically backs up data and transfers the backup to multiple external data centers.

Disaster Tolerance backup can still put data in a safe state when a serious problem occurs in the primary data center where redis runs and generates snapshots.

Because many redis users are entrepreneurs who don't have a lot of money to waste, the following describes some practical and inexpensive disaster recovery backup methods:

• Amazon S3 and other services similar to S3 are a good place to build a disaster backup system. The simplest way is to encrypt your RDB backup every hour or every day and send it to S3. Data Encryption can be implemented throughgpg -cCommand to complete (symmetric encryption mode ). Remember to put your password in a few different and secure places (for example, you can copy the password to the most important person in your organization ). Using multiple storage services to save data files can improve data security.
• You can use SCP to transmit snapshots (SSH components ). The following is a simple and secure transfer method: Buy a vps far away from your data center, install SSH, and create an SSH client key without a password, add the key to the VPs authorized_keys file, so that you can send the snapshot backup file to the VPs. To achieve the best data security, at least one VPs should be purchased from two different providers for Data Disaster Recovery Backup.

It should be noted that such disaster recovery systems are easy to fail if they are not carefully processed.

At the minimum, you should check whether the volume of the uploaded backup file is the same as that of the original snapshot file after the file is transferred. If you are using VPs, you can check whether the file is completely transmitted by comparing the sha1 checksum of the file.

In addition, you also need an independent alarm system to notify you when the transfer responsible for transferring backup files fails.

Redis persistence)