Apsaradb for redis-disk persistence and disaster tolerance backup

Preface

In the previous blog, the blog introduced the use cases of redis data types and the correct posture of redis distributed locks. We know that once redis is restarted, all data in redis will be lost. This blog introduces the persistence of redis disks.

 

Redis persistence

　　Achieve persistence by taking snapshots of redis at intervals

RDB persistence

Advantages: 1. It has little impact on redis performance.

2. When the dataset is large, the restoration speed is faster than Aof.

3. RDB is a very compact single file that can be easily transferred to a third-party Data Center (Amazon S3) for future disaster recovery.

Disadvantages: 1. Because of the snapshot persistence method of RDB, once a crash occurs, you may lose data for several minutes.

2. RDB needs to fork a sub-process to save the dataset to the disk. Therefore, when the dataset is large, redis will not respond to the client within milliseconds.

Configuration File: In the redis. conf file, there is such a paragraph.

################################ SNAPSHOTTING  ################################## Save the DB on disk:##   save <seconds> <changes>##   Will save the DB if both the given number of seconds and the given#   number of write operations against the DB occurred.##   In the example below the behaviour will be to save:#   after 900 sec (15 min) if at least 1 key changed#   after 300 sec (5 min) if at least 10 keys changed#   after 60 sec if at least 10000 keys changed##   Note: you can disable saving completely by commenting out all "save" lines.##   It is also possible to remove all the previously configured save#   points by adding a save directive with a single empty string argument#   like in the following example:##   save ""save 900 1save 300 10save 60 10000

In three policies, RDB executes snapshots to store data on disks. For example, save 60 10000 means that "at least 1000 keys are modified in 60 seconds", which automatically saves the dataset once.

Notice: When redis is just started, you use redis-CLI to store a piece of data into redis, and you will find that dump. RDB is not generated. This is because at this time there are no more than three policies in the group, so the bgsave command will not be executed. At this time, either you wait 900 seconds, or you enter redis-CLI, and actively execute the bgsave command.

This will generate dump. RDB.

Method of work: When data persistence is required, the operation is performed.

1. redis calls forks and has both parent and child processes. (This process is blocked)

2. The child process writes the dataset to a temporary RDB file.

3. redis replaces the old RDB file with the new RDB file and deletes the old RDB file.

 

Aof persistence

Persistence is achieved by appending redis write operations to the aof File

Advantages: 1. aof has three policies: fsync (fully dependent on the system, with performance close to RDB), fsync per second, and fsync (full data synchronization, but poor performance ). Fsync is used by default every second. This policy can only lose data in one second even if redis goes down. This ensures both performance and real-time data.

2. When writing a command to the aof file, the complete command is not written due to some problems (the disk is full or the write is down, you can use the redis-check-Aof command to fix these problems.

3. When the aof file is too large, the aof file will be automatically overwritten. (For example, if we only have one hundred incr count times, we will append the command 100 times to the aof file. After rewriting, there will be only one command similar to set count 100 ).

4. The aof file stores all write operations performed on redis in an orderly manner. These write operations are saved in the format of the redis protocol, which is very easy to understand. Therefore, when we perform some misoperations (flushall), we can also modify the aof file to repair the data.

Disadvantages: 1. When the dataset is the same, the volume is larger than that of RDB.

2. Unless fsync is used, the aof performance is slower than RDB.

Configuration File: view the configuration notes in the redis. conf file. The main configuration is as follows:

Appendonly yes whether to enable aof persistent appendfilename "appendonly. aof" aof file name

# Appendfsync always: The aof file appended to each write
Appendfsync everysec synchronizes aof files every second
# Appendfsync no does not synchronize aof files, fully dependent on the Operating System

The following are some other parameter configurations. For details, you can go to the official website to download a corresponding version of The redis. conf file.

Working principle: Like snapshot creation in RDB, aof rewrite uses the redis write-time replication mechanism cleverly:

1. redis executes fork () and has both parent and child processes.

2. The sub-process writes the new aof command to the temporary file.

3. for all newly executed write commands, redis puts them into the memory cache while writing them into the existing aof file, even if redis goes down during aof rewriting, the existing aof files are secure.

4. When the child process completes aof rewrite, it will send a signal to the parent process. After receiving the signal, all commands in the memory cache are appended to the new aof file.

5. redis replaces the original aof with the new Aof.

 

How to choose which persistence to use

1. If you have high data security requirements, we recommend that you use both of them.

2. If you can afford data loss for several minutes, you can use RDB

3. Use aof

Notice: the official redis website has a saying: for various reasons mentioned above, we may integrate aof and RDB into a single persistence model in the future. Maybe soon, we don't have to worry about what persistence to use.

 

Switch from RDB to aof

1. Create a backup for the existing RDB file and put the backup in a safe place.

2. When redis-CLI: config set appendonly Yes executes this command, redis will be blocked, knowing that the aof file is created completely, and then the new write command will be appended to the new aof file.

3. redis-CLI: config set save "" is optional to disable RDB persistence, because you can enable both the RDB and aof persistence functions at the same time.

4. Remember to synchronize the previous changes to the redis. conf configuration file. Otherwise, the configuration set in config set will become invalid after redis is restarted.

 

Back up redis data and disaster recovery

Remember: Make sure that your data is backed up completely. Disk faults, node failures, and other such problems may make your data disappear, and it is very dangerous not to back up your data.

Because of the working principle of RDB, redis is very friendly for data backup. At any time, copying RDB files is absolutely secure.

1. Create a scheduled task (cron job) to back up an RDB file to a folder every hour and back up an RDB file to another folder every day.

2. Make sure that all snapshots are backed up with corresponding date and time information. Use the find command to delete expired snapshots each time you execute a scheduled task Script: for example, you can retain snapshots per hour in the last 48 hours, and snapshots per day in the last one or two months.

3. at least once a day, back up RDB outside your data center, or at least to the physical machine on which you run the redis server.

After the data is backed up, you can store the backed up data in a third-party data center for disaster tolerance.

Apsaradb for redis-disk persistence and disaster tolerance backup