Redis data persistence, redis data

Redis data persistence, redis data
In general, there are two persistence solutions: RDB and AOF.
The RDB method creates a time point-based snapshot for the dataset at a certain interval.
The AOF method records the write operations received by the Server to log files. When the Server restarts, It replays these write operations to recreate the dataset. This method is similar to the statement-based binlog in MySQL. When the log size increases, Redis can re-write the log in the background.
If you only want data to exist during Server running, you can disable the two persistence schemes. You can enable both AOF and RDB data persistence solutions in the same Redis instance. In this case, when Redis restarts, The AOF file will be used to reconstruct the original dataset, because the AOF method can guarantee data integrity to the maximum extent in RDB mode.

Advantages and disadvantages of the two-minute solution
Advantages of RDB
RDB is a compact copy of Redis Data Set Based on time points, and is very suitable for backup scenarios. For example, you can archive a small RDB file every hour, archive a large RDB file every day, and archive a larger RDB file every month. In this way, you can select different backup versions to recover data when necessary.
RDB is ideal for disaster recovery because it is a compact file that is easy to transmit to a remote data center or Amazon S3.
The RDB mode has low overhead. In this mode, the Redis parent process only needs to open up a sub-process to do the rest.
Compared with AOF, RDB can restore data faster when the dataset is large.

Disadvantages of RDB
RDB is not the best solution if you want to avoid data loss when Redis stops working (such as unexpected power outages. For example, a snapshot is usually created every five minutes or longer. If Redis is not properly disabled, data in the last few minutes may be lost.
The RDB method often calls the fork () function to open up sub-processes for persistence. When the dataset is large and the CPU performance is not strong enough, the fork () call may take a lot of time, which may cause Redis to fail to serve the clients in several milliseconds or even one second. AOF also needs to call fork (), but can adjust the frequency of rewriting logs without affecting data persistence.

AOF advantages
When using the AOF method, Redis persistence is more reliable: There are three different fsync policies available: no fsync at all, fsync every second, and fsync at every query. The default value is fsync every second. At this time, the write performance is still good, And the write operation may be lost for one second in the worst case.
AOF logs are generated by append only. Therefore, there are no random access problems or damages caused by unexpected power outages. Even if for some reason (such as full disk) logs end with a half-written command, you can still use the redis-check-aof tool to quickly fix the problem.
When the AOF log grows, Redis can automatically rewrite the AOF log in the background. It is completely secure to re-write logs when Redis continues to append the old AOF log files. Redis uses the minimum command that can reconstruct the current dataset to generate a new log file. Once the new log file is created, Redis starts to append logs to the new log file.
AOF log format is easy to understand and parse. This is useful in some scenarios. For example, you cannot use the FLUSHALL command to clear all the data, and AOF logs are not overwritten, you can simply stop the Redis Server and remove the last FLUSHALL command in the log to restart the Redis Server to recover data.

AOF disadvantages
The same dataset AOF file is much larger than the RDB file.
Based on the fsync method, AOF may be much slower than RDB. When no fsync at all is used, the performance of AOF is basically the same as that of RDB. When fsync every second is used, the performance is decreased but still high, and the performance is low when fsync at every query is used. However, the RDB method can minimize the latency at high loads.
Some specific commands may have bugs, so that the same dataset cannot be rebuilt when AOF logs are reloaded. Such a bugs is very rare and has been fully tested through the test suite. This type of bugs is almost impossible for RDB. More clearly: Redis AOF incrementally updates the existing status and RDB snapshots are re-created each time. In terms of concept, the RDB method is more robust. However, pay attention to two points: each time the AOF log is overwritten by Redis, the log is re-generated by the actual data containing the dataset, compared with the append AOF file method, this method can effectively reduce the probability of bugs appearing. in actual application scenarios, no user reports about AOF damage have been received.

How to Choose persistence mode?
Depends on the Specific Application Scenario. Generally, two methods can be used at the same time. If you are concerned about data but can still tolerate data loss for several minutes, you can simply use the RDB method. Many users only use the AOF method. This method is not recommended. On the one hand, creating an RDB snapshot at a certain interval is an excellent way to create a data backup and quickly restore data, on the one hand, you can avoid the possible bugs in the AOF mode. For the above reasons, the AOF and RDB methods may be combined into one in the future.

RDB persistence settings
By default, Redis creates a data snapshot named dump. rdb in binary format on the disk. You can create a snapshot every N seconds through the configuration file and at least M changes on the dataset, whether to compress the data, the snapshot name, and the working directory where the snapshot is stored. The default configuration of redis 2.4.10 is as follows:

# Create a snapshot after 900 seconds and at least one key changes save 900 1 # create a snapshot after 300 seconds and at least 10 keys change save 300 10 #60 seconds later and at least create snapshot save 60 10000 When 10000 keys change # You can disable RDB persistence by commenting on all rows starting with save # compress data during snapshot creation rdbcompression yes # snapshot name dbfilename dump. rdb # directory for storing snapshots (AOF files will also be stored in this directory) dir/var/lib/redis/

For more information about configuration parameters, see the description in redis. conf.

In addition to setting through the configuration file, you can also manually execute commands to create snapshots.
The SAVE command executes a synchronization operation to SAVE snapshots of all data in the instance as RDB files. Generally, the SAVE command is not directly used in the production environment, because all client requests are blocked and can be replaced by the BGSAVE command. Create a data snapshot in the BGSAVE background. The status code of the name execution result is returned immediately. Redis opens up a sub-process, the parent process continues the corresponding client request, the sub-process saves the database to the disk and then exits. The client can check whether the operation is successful by executing the LASTSAVE command.

Workflow for creating RDB snapshots
When Redis needs to dump the dataset to the disk, the following process is performed:
Redis forks a sub-process;
Child processes write data sets to temporary RDB files;
The child process writes a new RDB file and replaces the old RDB file.
This method enables Redis to take advantage of the copy-on-write mechanism.

AOF persistence settings
The Snapshot persistence method is not very reliable. When the computer running Redis stops working, accidentally loses power, or accidentally kills the Redis process, the data recently written into Redis will be lost. This may not be a problem for some applications, but it is not an ideal choice for snapshot methods in scenarios with high persistence requirements. AOF file is an alternative solution for maximizing data persistence. Similarly, you can use the configuration file to open and close AOF:

# Disable AOFappendonly no # enable AOFappendonly yes

After the value of appendonly is set to yes, commands for changing the dataset received by Redis are appended to the AOF file every time. After Redis is restarted, The AOF file is replayed to recreate the data.

You can also configure the AOF file name, the frequency of calling fsync, the behavior of calling fsync, and rewrite the AOF condition through the configuration file. The default configuration of redis 2.4.10 is as follows:

# Default AOF file name appendfilename appendonly. aof # Call fsync every second to refresh data to the disk appendfsync everysec # When the BGSAVE or BGREWRITEAOF command is being executed, the fsync () call in the main process is not blocked (no by default, when there is a latency problem, you need to adjust it to yes) no-appendfsync-on-rewrite no # When AOF increases by 100% and reaches 64 mb, the system will automatically rewrite AOFauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64 mb

For more information about the parameters, see redis. conf.

Notes
Log Rewriting
As Redis receives more commands, the AOF file will become larger and larger. Redis supports the log rewriting feature. you can reconstruct the AOF file in the background without affecting the response client. After BGREWRITEAOF is executed in Redis, Redis uses the minimum command required to build the dataset to reconstruct the log file. BGREWRITEAOF needs to be run manually in Redis2.2. Redis2.2 starts to support automatic triggering of log rewriting.

Log rewriting also uses the copy-on-write mechanism. The process is roughly as follows:
Redis opens up a sub-process;
The child process writes a new AOF file in the temporary file;
The parent process caches all new changes in memory (and the new changes are written to the old AOF, which is safe even if the rewrite operation fails );
After the temporary AOF is rewritten by the child process, the parent process receives a signal and appends the buffer changes in memory to the end of the temporary file generated by the child process;
Redis renames the file and replaces the old file with the new file and begins to append new data to the new file.

Fsync call mode
This mode determines how often Redis refreshes data to the disk. There are three options:
No fsync at all is determined by the operating system when data is flushed. The fastest but least secure.
Fsync every second refresh every second. Fast enough to lose up to one second of data.
Fsync at every query record a new command to AOF and then fl the data to the disk. Slowest but safest.
The default policy (also the Default policy) is fsync every second.

AOF countermeasures for damage
If the Server crashes during AOF file writing, the AOF file may be damaged and cannot be loaded by Redis. You can perform the following steps to fix the problem:
Creates an AOF file backup;
Use the redis-check-aof tool to repair the original AOF file;
$ Redis-check-aof -- fix
Use diff-u to check the similarities and differences between the backup file and the repaired file (optional );
Use the repaired AOF file to restart Redis.

How does one transition from RDB to AOF persistence?
Redis> = 2.2
Create a backup of the latest RDB file;
Save the backup to a secure location;
Run the following command;
$ Redis-cli config set appendonly yes
$ Redis-cli config set save "" (optional, if the RDB and AOF methods are not executed, they coexist)
Check that the database contains the same keys;
Confirm that the write operation is correctly appended to the AOF file.
Note: Remember to modify the corresponding configuration in redis. conf to avoid the loss of Configuration updates by running commands after Redis Server is restarted, and then re-use the configuration in the old configuration file.

Redis2.0
Create a backup of the latest RDB file;
Store backups in a safe location;
Stop all write operations on the database;
Initiate the redis-cli bgrewriteaof command to create the AOF file;
When the AOF file is generated, stop the Redis Server;
Edit redis. conf to enable AOF persistence;
Restart the Redis Server;
Check that the database contains the same keys;
Confirm that the write operation is correctly appended to the AOF file.

Interaction between AOF and RDB
Redis2.4 and later versions will ensure that AOF rewrite is not triggered during RDB snapshot creation or BGSAVE is not allowed during AOF rewrite, to avoid the Redis background process performing heavy disk I/O operations at the same time.
When you create an RDB snapshot, the server immediately responds to an OK status code for the log rewriting operation explicitly initiated by the user using BGREWRITEAOF to notify the user that the operation will be executed again, this operation is only executed when the snapshot is created.
When both AOF and RDB are used, the AOF file is used to reconstruct the original dataset after Redis is restarted.

Back up Redis data
Make sure to back up data to prevent accidental loss. Redis is backup-friendly and can copy RDB files when the database is running. Recommended backup solution:
Create a cron job create an RDB snapshot every hour in one directory create an RDB snapshot every day in another directory;
When a cron job runs, run the "find" command to clear out the obsolete RDB snapshot files (you can mark the time and data contained in the snapshot hit );
Ensure that the RDB snapshot is transferred to an external data center or at least one machine outside the physical machine running the Redis instance (at least once a day ).

Disaster recovery
Disaster recovery and data backup in Redis are basically the same process. You can consider distributing backups to different remote data centers to avoid data loss to the maximum extent. Several low-cost disaster recovery plans:
Amazon S3 or other similar services are a good choice. RDB snapshots of every hour can be encrypted (gpg-c encryption can be used) transmitted to S3 every day. Ensure that passwords are stored in different security areas. We recommend that you use different storage services to improve data security.
Use the SCP command to transmit the snapshot to the remote server. The simplest and safest way: obtain a small remote VPS, install ssh on it, generate an ssh client key without a password, and add it to the VPS authorized_keys file, then you can use SCP to transmit the backup data to the VPS. We recommend two different VPS to improve security.
It should be noted that the integrity and correctness of the file must be verified after the file is transferred. It can be verified by MD5 or SHA1. In addition, you need to set up an alarm system to inform you of backup transmission problems in a timely manner.