Redis persistence, redis
Web programmer blog: http://blog.csdn.net/thinkercode
Redis persistence Mechanism
Redis is a memory database that supports persistence. That is to say, redis often needs to synchronize data in the memory to the disk to ensure persistence.
Redis provides multiple levels of Persistence:
- RDB persistence allows you to generate a point-in-time snapshot of a dataset within a specified interval ).
- AOF records all write operation commands executed by the server persistently, and re-executes these commands to restore the dataset when the server starts. All commands in the AOF file are saved in the format of the Redis protocol. The new commands are appended to the end of the file. Redis can also rewrite the AOF file in the background so that the size of the AOF file does not exceed the actual size required to save the dataset status.
- Redis can also use both AOF persistence and RDB persistence. In this case, when Redis restarts, it will first use the AOF file to restore the data set, because the data set saved in the AOF file is generally more complete than the data set saved in the RDB file.
- You can even disable the persistence function so that data exists only when the server is running.
Advantages and disadvantages of RDB
Advantages of RDB
- RDB is a compact file that stores Redis data sets at a certain point in time. This type of file is very suitable for backup: for example, you can back up the RDB file every hour within the last 24 hours, and back up an RDB file every day of each month. In this way, the dataset can be restored to different versions at any time even if a problem occurs.
- RDB is very suitable for disaster recovery (disaster recovery): it only has one file, and the content is very compact, it can be (encrypted) transmitted to another data center, or Amazon S3.
- RDB can maximize the performance of Redis: The only thing the parent process has to do when saving the RDB file is to fork a sub-process, and then the sub-process will process all the subsequent save work, the parent process does not need to perform any disk I/O operations.
- RDB is faster than AOF in recovering big data sets.
Disadvantages of RDB
- If you need to avoid data loss when the server fails, RDB is not suitable for you. Although Redis allows you to set different save points to control the frequency of saving RDB files, because the RDB file needs to save the status of the entire dataset, therefore, it is not an easy operation. Therefore, you may save the RDB file at least five minutes. In this case, you may lose data for several minutes in the event of a fault or shutdown.
- Every time you save the RDB, Redis has to fork () A sub-process, and the sub-process is used for actual persistence. When the dataset is large, fork () may take a very long time, causing the server to stop processing the client within a certain millisecond. If the dataset is large and the CPU time is very short, the stop time may even be up to one second. Although AOF rewrite also requires fork (), no matter how long the execution interval of AOF rewrite is, data durability will not suffer any loss.
AOF advantages and disadvantages
AOF advantages
- Using AOF persistence will make Redis very durable (much more durable): You can set different fsync policies, such as no fsync, fsync every second, or fsync every time you execute the write command. The default policy of AOF is fsync every second. In this configuration, Redis can still maintain good performance, and even if a fault is down, at most one second of data will be lost (fsync will be executed in the background thread, so the main thread can continue to process command requests ).
- The AOF file is an append-only log file. Therefore, you do not need to perform seek for writing AOF files, even if the log contains incomplete commands for some reason (for example, when the disk is full, the write is stopped midway through, and so on ), the redis-check-aof tool can also easily fix this problem.
- Redis can automatically rewrite AOF in the background when the AOF file size is too large: The New AOF file after rewriting contains the minimum command set required to restore the current dataset. The entire rewrite operation is absolutely safe, because Redis will continue to append the command to the existing AOF file during the creation of the AOF file, even if there is a shutdown during the rewrite process, existing AOF files will not be lost. Once the new AOF file is created, Redis will switch from the old AOF file to the new AOF file and start appending the new AOF file.
- The AOF file stores all write operations performed on the database in an orderly manner. These write operations are saved in the format of the Redis protocol. Therefore, the content of the AOF file is very easy to understand, it is also easy to analyze files (parse. The export (export) AOF file is also very simple: for example, if you accidentally execute the FLUSHALL command, but as long as the AOF file is not overwritten, you only need to stop the server, remove the FLUSHALL command at the end of the AOF file and restart Redis to restore the data set to the State before the execution of FLUSHALL.
Disadvantages of AOF
- For the same dataset, the size of the AOF file is usually larger than that of the RDB file.
- Based on the fsync policy, AOF may be slower than RDB. In general, the performance of fsync per second is still very high, and disabling fsync can make AOF as fast as RDB, even under high load. However, when processing massive write loads, RDB can provide a more guaranteed maximum latency (latency ).
- AOF has encountered such a bug in the past: Due to some commands, when the AOF file is re-loaded, it cannot restore the dataset to its original state when it is saved. (For example, the blocking command BRPOPLPUSH has caused such a bug .) Tests are added to the test suite for this situation: they automatically generate random and complex datasets and ensure that everything works by re-loading the data. Although this bug is not common in AOF files, RDB is almost impossible to produce this bug.
RDB and AOF, which one should I use?
In general, two persistence functions should be used at the same time to achieve data security comparable to PostgreSQL.
If you are very concerned about your data, but still can withstand data loss within several minutes, you can only use RDB for persistence.
Many users only use AOF persistence, but this method is not recommended because it is very easy to back up databases by generating RDB snapshots on a regular basis, in addition, RDB can recover data sets faster than AOF. In addition, RDB can avoid the previous AOF program bugs.
Use RDB persistence policies
- Configure RDB persistence policies
save 900 1save 300 10save 60 10000dbfilename dump.rdbdir ./
By default, Redis saves the database snapshot in the binary file named dump. rdb. You can change the dbfilename and dir options to adjust the location of the snapshot file.
You can set the save policy of apsaradb for Redis to automatically save the dataset once the condition "the dataset has at least M changes in N seconds" is met.
You can also manually ask Redis to SAVE the dataset by calling SAVE or BGSAVE.
[root@localhost bin]# ./redis-server restart
Snapshot Operation Method
When Redis needs to save the dump. rdb file, the server performs the following operations:
Use AOF persistence Policy
- Configure the AOF persistence Policy
appendonly yes
From now on, every time Redis executes a command to change the dataset (such as SET), this command will be appended to the end of the AOF file.
In this way, when Redis is restarted, the program can re-execute the commands in the AOF file to recreate the dataset.
[root@localhost bin]# ./redis-server restart
AOF Operation Method
Like creating snapshots in RDB, AOF rewrite uses the write-time replication mechanism cleverly.
The following describes how to rewrite AOF:
Use the RDB and AOF dual persistence policies
- Configure persistence Policy
save 900 1save 300 10save 60 10000dbfilename dump.rdbdir ./appendonly yes
[root@localhost bin]# ./redis-server restart
- Interaction between RDB and AOF
When Redis is started, if both RDB persistence and AOF persistence are enabled, the program will first use the AOF file to restore the dataset, because the data stored in the AOF file is usually the most complete.
BGREWRITEAOF cannot be executed during BGSAVE execution. Likewise, BGSAVE cannot be executed during BGREWRITEAOF execution. This prevents two Redis background processes from simultaneously performing a large number of I/O operations on the disk.
If BGSAVE is being executed and the user explicitly calls the BGREWRITEAOF command, the server will return an OK State to the user and inform the user that BGREWRITEAOF has been scheduled to be executed: Once BGSAVE is completed, BGREWRITEAOF will be officially started.
AOF persistence mechanism problems
- AOF rewrite
Because AOF constantly appends commands to the end of a file, the size of the AOF file increases as the number of write commands increases. For example, if you call INCR for a counter for 100 times, you only need to use 100 entries in the AOF file to save the current value of the counter ). However, in fact, only one SET command is enough to save the current value of the counter, and the other 99 records are actually redundant. To handle this situation, Redis supports an interesting feature: you can rebuild the AOF file without interrupting the service client ). Run the BGREWRITEAOF command. Redis will generate a new AOF file, which contains the minimum command required to reconstruct the current dataset. Redis can automatically trigger AOF rewrite.
- What is the durability of AOF?
You can configure how often Redis fsync data to the disk.
There are three options:
- Execute fsync every time a new command is appended to the AOF file: It is very slow and secure.
- Fsync once per second: fast enough (similar to RDB persistence), and only one second of data will be lost in the event of a fault.
- Never fsync: submit the data to the operating system for processing. Faster and less secure options.
The recommended (and also the default) method is fsync once per second. This fsync policy can take both speed and security into account. The fsync policy is always very slow in actual use, even after Redis 2.0 has improved the related program, it is still so-frequent calls to fsync are doomed to this strategy cannot get faster.
- What if an error occurs in the AOF file?
The server may be shut down when the program is writing AOF files. If the AOF file is faulty, Redis will refuse to load the AOF file when it is restarted, this ensures that data consistency is not damaged.
In this case, you can use the following methods to fix the AOF file:
How to switch from RDB persistence to AOF persistence
You can switch from RDB to AOF without restarting:
CONFIG SET appendonly yesCONFIG SET save ""
Make sure that the number of database keys does not change after the command is executed.
Make sure that the write command is correctly appended to the end of the AOF file.
The AOF function is enabled for the First Command executed in step 3: Redis will block until the initial AOF file creation is complete, and Redis will continue to process the command request, and start to append the write command to the end of the AOF file.
The second command executed in step 3 is used to disable the RDB function. This step is optional. You can also use both the RDB and AOF persistence functions if you want.
Don't forget to enable the AOF function in redis. conf! Otherwise, after the server is restarted, the configuration previously SET through config set will be forgotten, and the program will start the server according to the original configuration.
Back up Redis data
Disk faults, node failures, and other such problems may make your data disappear, and it is very dangerous not to back up your data.
Redis is very friendly for data backup. You can copy the RDB file when the server is running: Once the RDB file is created, it will not be modified. When the server creates a new RDB file, it first saves the file content in a temporary file. When the temporary file is written, the program uses rename (2) replace the original RDB file with a temporary file atomically. This means that, at any time, copying RDB files is absolutely safe.
The following are official suggestions:
- Create a scheduled job to back up an RDB file to a folder every hour and back up an RDB file to another folder every day.
- Make sure that all snapshots are backed up with corresponding date and time information. Use the find command to delete expired snapshots each time you execute a scheduled task Script: for example, you can retain snapshots per hour in the last 48 hours, and snapshots per day in the last one or two months.
- Back up RDB to external data centers at least once a day, or at least to physical machines running your Redis server.
