In general there are two types of persistence scenarios: Rdb and AoF
The Rdb method creates a point-in-time snapshot of the dataset at certain intervals.
The AoF method records the write operations received by the server to the log files and rebuilds the datasets by replaying them when the server restarts. This method is similar to the Binlog in MySQL based on the statement format. Redis can rewrite logs in the background when the log becomes large.
Two persistence scenarios can be disabled if only the expected data exists during server runtime. It is also possible to turn on the data persistence scheme for both the AOF and RDB modes in the same Redis instance. In this case, the AoF file will be used to reconstruct the original data set when the Redis restarts, because the AOF approach maximizes data integrity, called the RDB approach.
advantages and disadvantages of the two-clock program
RDB Benefits
The RDB is a compact, point-in-time copy of the Redis dataset and is ideal for backup scenarios. For example, every hour to the Rdb file to do a small archive, the Rdb file to do a large archive every day, the Rdb file once a month to do a larger archive. This allows you to select a different backup version for data recovery at the necessary time.
Because it is a compact file that is easy to transfer to a remote data center or Amazon S3, the RDB is ideal for disaster recovery.
The RDB approach is less expensive, and in that way the Redis parent process is only going to open up a subprocess to do the rest.
An RDB can recover faster than a aof when the data set is large.
Rdb Disadvantages
An RDB is not the best solution if it is necessary to ensure that data is not lost when Redis stops working, such as accidental power outages. For example, a snapshot is typically created every 5 minutes or more, and if Redis is not properly closed, the last few minutes of data may be lost.
The Rdb method often calls the fork () function to open a child process for persistence. A fork () call can be time-consuming when the dataset is large and CPU performance is not strong enough to cause Redis to fail to service clients in milliseconds or even seconds. AOF also needs to call fork () but can adjust the frequency of rewrite logs without affecting the persistence of the data.
AOF Advantages
Redis persistence is more reliable when using aof: There are three different fsync strategies to choose from: No fsync at all, Fsync every second, fsync at every query. The default is fsync every second write performance is still good at this time, and in the worst case, a second write operation may be lost.
The AOF log is a log generated by the Append only method, so there is no random access problem and damage caused by accidental power outages. Even if for some reason (such as disk full) The log ends with a half-written command, you can still use the redis-check-aof tool to quickly fix it.
When the aof log becomes larger, Redis can automatically rewrite the aof log in the background. Rewriting the log is completely safe when Redis continues to append the old aof log file. Redis leverages the fewest commands that can reconstruct the current dataset to produce a completely new log file, and once the new log file is created, Redis begins appending the log to the new log file.
The AOF log format is easy to understand and easy to parse. This is useful in some scenarios. For example, if you use the Flushall command to clear all the data, and the AOF log does not have a rewrite operation, you can simply restart Redis server to recover data by stopping the last Flushall command in the Redis server removal log.
aof Disadvantages
The same dataset AoF file is much larger than the Rdb file.
Depending on the Fsync method used, the aof may be much slower than the RDB. The performance of the aof is essentially flat with the RDB when using the No fsync at all, and performance decreases while using the fsync every second and is still high, with low performance when using Fsync at every query. The Rdb method, however, can ensure that the delay is as small as possible under high load conditions.
Some specific commands may have bugs that can cause the AOF log to be overloaded without rebuilding the exact same data set. Such bugs are very, very rare, and have been fully tested by the test suite. This type of bugs is almost impossible for an RDB. More clearly: The Redis aof incremental update is in the existing state and the RDB snapshot is re-created each time, conceptually, the RDB way is more robust. However, there are two points to note: each time the AOF log is rewritten by Redis, the log is regenerated by the actual data containing the dataset, which effectively reduces the probability of bugs occurrence compared to the way the aof file is appended; no user reports of aof damage have been received in real-world scenarios.
How do I choose a persistence method?
Depending on the application scenario, in general, both methods can be used simultaneously. If you are more concerned with data but still can tolerate a few minutes of data loss, you can simply use the Rdb method. There are many users who only use the AoF method, and it is not recommended to create an RDB snapshot at a certain interval is an excellent way to create a data backup and recover data quickly, on the one hand, to avoid possible bugs in aof way. For these reasons, aof and RDB may be merged in the future.
RDB Persistence Settings
By default, Redis creates a binary-formatted data snapshot on disk that is named Dump.rdb. You can configure a profile to create a snapshot every n seconds and at least m changes on the dataset, whether to compress the data, the snapshot name, and the working directory where the snapshot is stored. The default configuration for Redis 2.4.10 is as follows:
Create a snapshot after the #900秒后且至少1个key发生变化时创建快照save 900 1#300 seconds and at least 10 keys change when save 300 10#60 seconds and at least 10,000 keys are changed when save 60 10000# You can disable RDB persistence by annotating all rows that start with Save rdbcompression yes# snapshot name dbfilename dump.rdb# The directory where the snapshot is stored (the AoF file will also be stored in this directory) dir/var/ lib/redis/
For more information about configuration parameters, refer to the instructions in redis.conf.
You can create a snapshot by manually executing commands in addition to setting it through a configuration file
The Save command performs a synchronous operation that holds a snapshot of all the data in the instance as an RDB file. The Save command is generally not used directly in the production environment because it blocks all client requests and can be replaced with the Bgsave command. Bgsave Create a data snapshot in the background. The status code of the named execution result is returned immediately. Redis opens up a child process, and the parent process continues the corresponding client request, and the child process saves the DB to disk and exits. The client can check whether the operation was successful by executing the Lastsave command.
workflow for creating an RDB snapshot
The following procedures are performed when Redis requires a dump dataset to disk:
Redis forks a child process;
The child process writes the dataset to a temporary RDB file;
After the child process finishes writing the new Rdb file, replace the old Rdb file.
This approach enables Redis to leverage the benefits of the copy-on-write mechanism.
aof Persistence Settings
The persistence of snapshots is not very reliable, and the data that was recently written to Redis will be lost when the computer running Redis stops working, accidentally loses power, and accidentally kills the Redis process. This may not be a problem for some applications, but it is not an ideal choice for scenario snapshots that are very demanding for persistence. The AoF file is an alternative to maximizing data persistence. Similarly, you can open and close aof by using a configuration file:
#关闭AOFappendonly no# Open Aofappendonly Yes
When AppendOnly is set to Yes, the command to change the dataset received by Redis will be appended to the AoF file each time. Restarting Redis will replay the aof file to reconstruct the data.
You can also configure aof file names through configuration files, the frequency of calls to Fsync, the behavior of calling Fsync, overriding aof conditions. The default configuration for Redis 2.4.10 is as follows:
#默认AOF文件名appendfilename appendonly.aof# calls Fsync refresh data to disk Appendfsync per second everysec# Fsync () calls in the main process are not blocked when the Bgsave or BGREWRITEAOF command is executing in the process (default is no, adjust to Yes when there is a delay problem) No-appendfsync-on-rewrite no# when aof growth rate is 100% And when it reaches 64MB, it begins to automatically rewrite Aofauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mb
The meaning of each parameter can be found in the detailed description in redis.conf.
A few notes
Log Rewriting
As Redis receives more commands, the aof file becomes larger. Redis supports the log rewriting feature, which allows the aof file to be refactored in the background without affecting the response client. After performing bgrewriteaof in Redis, Redis will refactor the log files using the fewest commands required to build the dataset. Redis2.2 need to be run manually bgrewriteaof,redis2.2 start support automatically triggering log rewriting.
Log rewriting also uses the copy-on-write mechanism, which is roughly the following:
Redis opens up a sub-process;
The child process writes the new AoF file in the temporary file;
The parent process caches all new changes in memory (while the new changes are written to the old AoF, which is also safe even if the rewrite fails);
The child process rewrites the temporary AOF stepfather process receives a signal and appends the buffered changes in memory to the end of the temporary file produced by the child process;
Redis file Rename replaces the old file with a new file and begins appending new data to the new file.
Fsync Call Pattern
This mode determines how often redis refreshes data to disk, with three options available:
No fsync at all is determined by the operating system when it is time to brush the data. Fastest but least secure.
Fsync every second refreshed once per second. Fast enough to lose up to one second of data.
Fsync at every query each time a new command is recorded to aof the data is brushed once to disk. The slowest but safest.
The default policy (also the default policy) is fsync every second
countermeasures for AOF damage
If the server crashes while writing the aof file, it may cause the aof file to become corrupted and not be loaded by Redis. This can be fixed by the following steps:
Create a backup of a aof file;
Use the Redis-check-aof tool to repair the original aof file;
$ redis-check-aof--fix
Use Diff-u to check the similarities and differences between backup files and repaired files (optional steps);
Restart Redis with the repaired aof file.
How do I convert from RDB persistence to aof persistence?
when Redis >=2.2
Create a backup of the most recent RDB file;
Save the backup in a secure location;
To initiate the following order;
$redis-cli Config set appendonly Yes
$redis-cli config Set save "" (optional, if the RDB is not executed and the AOF mode will coexist)
Confirm that the database contains the same keys;
Confirm that the write operation was correctly appended to the AoF file.
Note: Remember to modify the corresponding configuration in the redis.conf to avoid the loss of configuration updates through the command after Redis server restarts and reuse the configuration in the old configuration file.
when Redis2.0
Create a backup of the most recent RDB file;
Store the backup in a secure location;
Stop all write operations on the database;
Launch the REDIS-CLI bgrewriteaof command to create the aof file;
Stop Redis Server After the aof file is generated;
Edit redis.conf open aof persistence;
Restart Redis Server;
Confirm that the database contains the same keys;
Confirm that the write operation was correctly appended to the AoF file.
the interaction between AOF and Rdb
Versions above Redis2.4 Ensure that aof overrides are not triggered when an RDB snapshot is created, or bgsave operations are not allowed when aof overrides, to avoid heavy disk I/O operations at the same time for Redis daemon processes.
When an Rdb snapshot is created, an explicitly initiated log rewrite operation for the user using Bgrewriteaof server responds immediately to an OK status code informing the user that the operation will be executed back, when and only when the snapshot creation is complete, the rewrite operation begins to execute.
In cases where both aof and RDB are used, the Redis restart takes precedence over the aof file to refactor the original dataset.
Backing up Redis data
Be sure to do a good job of data backup to avoid accidental loss. Redis is backup-friendly and can copy RDB files while the database is running. Recommended Backup scenarios:
Create a cron job create an RDB snapshot one time per hour in a directory an RDB snapshot is created daily in another directory;
The cron job uses the Find command every time it runs to ensure that obsolete rdb snapshot files are cleaned out (can be tagged by including data and time information in the snapshot hit);
Ensure that the Rdb snapshot is transferred to an external datacenter or at least a machine outside the physical machine running the Redis instance (at least once a day).
Disaster Recovery
Disaster recovery and data backup are essentially the same process in Redis. Consider distributing backups to different remote datacenters to minimize data loss. Several low-cost disaster recovery plans:
Amazon S3 or other similar services is a good choice. You can transfer an RDB snapshot every hour per day in an encrypted manner (which can be encrypted using gpg-c) to S3. Be sure to store your password in a different, secure place. It is recommended to use different storage services to improve data security.
Use the SCP command to transfer the snapshot to a remote server. The simplest and safest way: Get a small remote VPS, install SSH on it, generate a password-free SSH client key to add to the VPS's Authorized_keys file, and then use the SCP transfer to back up to the VPS. It is recommended to engage two different VPS to improve security.
It is important to note that the integrity of the file must be verified after the file transfer is complete. can be verified by MD5 or SHA1. In addition, a set of alarm system needs to be built, when the backup transmission occurs when the problem can be timely informed.
Redis Data Persistence
