Redis persistence mechanism

Redis has two storage methods, the default is the snapshot way, the implementation method is to periodically persist the memory snapshot (snapshot) to the hard disk, the disadvantage is that if the crash occurs after persistence, it will lose a piece of data. As a result of the perfectionist's push, the author added the AoF way. aof that is append only mode, the operation command is saved to the log file while writing the memory data, in a system with tens of thousands of concurrent changes, the command log is a very large amount of data, management and maintenance costs are very high, restore the rebuild time is very long, This leads to the loss of aof high availability intent. What's more, Redis is a memory data structure model, and all of the advantages are based on the efficient atomic operation of complex memory structures, so that aof is a very uncoordinated part.

In fact, the main purpose of AOF is data reliability and high availability, in Redis there is another way to achieve the purpose: Replication. Because of the high performance of Redis, there is virtually no delay in replication. This is achieved by preventing single points of failure and achieving high availability.

Good practice: Turn off the master persistence feature so that the data is only turned on when the 10th Slave server node is running AOF.

AoF rewrite

You can think, every write command generates a log, then the aof file is not very large?

The answer is yes, the AoF file will grow larger, so Redis provides a feature called AoF rewrite. Its function is to regenerate a copy of the AoF file, one record in the new AoF file is only once, and unlike an old file, multiple operations on the same value may be logged.

Its build process is similar to an RDB, and it also fork a process, traversing the data directly, and writing a new aof temporary file. In the process of writing a new file, all of the write logs are still written to the old aof file and are also recorded in the memory buffer. When the completion of the operation completes, logs from all buffers are written to the temporary file once. Then call the atomic Rename command to replace the old aof file with the new AoF file.

From the above process we can see that both RDB and AOF operations are sequential IO operations with high performance. At the same time, when the database is restored through the Rdb file or the AOF log, the sequential read data is loaded into memory. So it does not cause random reads of the disk.

AOF Reliability Settings

AoF is a write-file operation that is intended to write the operation log to disk, so it will also encounter the 5 processes we have described above for the write operation. So how high is the operational security of writing AOF? In fact, this can be set, in Redis in the AOF call write (2) write, when the call Fsync write it to disk, through the Appendfsync option to control, the following configuration method

Rdb and aof persistence vs. RDB mechanism advantages and slightly

RDB persistence refers to writing a snapshot of an in-memory dataset to disk within a specified interval of time. is also the default persistence method, which is to write the in-memory data in a snapshot to the binary file, the default file name is Dump.rdb.

You can automatically make snapshot persistence by configuring settings. We can configure Redis to take snapshots automatically if more than M key is modified in n seconds, the following is the default snapshot save configuration

   Save 1  #900秒内如果超过1个key被修改,    The snapshot is saved #300秒内容如超过10个key被修改, the snapshot is saved    60 10000

Advantage

• Once this is done, your entire Redis database will contain only one file, making it easy to back up. For example, you may not want to archive some data for 1 days.
• Easy Backup, we can easily move an RDB file to other storage media
• An RDB recovers a large data set faster than AOF.
• An RDB maximizes the performance of Redis: The only thing the parent process has to do when saving an Rdb file is to fork out a child process, and the child process will handle all subsequent save work, and the parent process does not have to perform any disk I/O operations.

Disadvantage

• If you need to avoid losing data when the server fails, the RDB is not for you. Although Redis allows you to set different savepoint (save point) to control how often an RDB file is saved, it is not an easy operation because the Rdb file needs to preserve the state of the entire dataset. Therefore, you may be able to save the RDB file at least 5 minutes. In this case, you may lose several minutes of data in the event of a failure.
• Each time you save an RDB, Redis will fork () out a subprocess and perform the actual persistence work by the child process. When the dataset is large, fork () can be time consuming, causing the server to stop processing the client in such milliseconds, and if the data set is huge and CPU time is very tight, the stop time may even be a full second. Although the AOF rewrite also requires fork (), there is no loss of data durability regardless of the length of the AOF rewrite execution interval.

AOF File Save Process

Redis appends each received write command to the file via the Write function (default is appendonly.aof).

When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands saved in the file. Of course, because the OS caches write modifications in the kernel, it may not be written to disk immediately. The persistence of this aof method is also likely to lose some of the modifications. But we can tell Redis through the configuration file that we want to force the OS to write to disk through the Fsync function. There are three ways to do this (the default is: Fsync once per second)

AppendOnly Yes              // enable aof persistence # Appendfsync      always// every time you receive a write command, it is forced to write to disk immediately, the slowest, But to ensure full persistence, it is not recommended to use appendfsync everysec     // Force write disk once per second, a good compromise in performance and persistence, recommended # Appendfsync No    // full reliance on OS, best performance, no guarantee of persistence

The AOF approach also poses another problem. Persistent files can become more and more large. For example, we call the INCR Test command 100 times, the file must save all 100 commands, in fact, 99 are redundant. Because you want to restore the state of the database, it is enough to save a set test 100 in the file.

In order to compress the aof persistence file. Redis provides the bgrewriteaof command. Receive this command Redis will use a snapshot-like method to save the in-memory data to a temporary file in the form of a command, and finally replace the original file. The specific process is as follows

• Redis Call Fork, now has parent-child two processes
• A child process writes a command to a temporary file to rebuild the state of a database based on a database snapshot in memory
• The parent process continues to process the client request, in addition to writing the write command to the original aof file. Cache the received write commands at the same time. This will ensure that if the child process rewrite fails, it will not be problematic.
• The child process signals the parent process when the child process writes the snapshot content to a temporary file that has been written to the command mode. The parent process then writes the cached write command to the temporary file as well.
• Now the parent process can replace the old aof file with the temporary file and rename it, and the subsequent write commands are appended to the new AoF file.

Note that the operation to rewrite the aof file does not read the old aof file, but instead overwrites the entire in-memory database content with a new aof file in the form of a command, which is a bit similar to a snapshot.

Advantage

• Using AOF persistence makes Redis very durable (much more durable): You can set different fsync policies, such as no fsync, one fsync per second, or Fsync each time you execute a write command. The default policy for AOF is Fsync once per second, in which Redis can still maintain good performance and, even if there is a failure, it will lose at most one second of data (Fsync will execute in a background thread, so the main thread can continue to work on the command request).

• The AOF file is an append-only log file (append only), so the write to the AOF file does not require seek, even if the log contains incomplete commands for some reason (such as a write-in disk full, write-down, etc.), The Redis-check-aof tool can also easily fix this problem.
  Redis can automatically override AOF in the background when the AOF file size becomes too large: The rewritten new AOF file contains the minimum set of commands required to restore the current dataset. The entire rewrite operation is absolutely secure, as Redis will continue to append commands to the existing AOF file during the creation of the new AOF file, and the existing AOF files will not be lost even if there is an outage during the rewrite. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and begins appending to the new AOF file.

• The AOF file preserves all write operations performed on the database in an orderly manner, and these writes are saved in the format of the Redis protocol, so the contents of the AOF file are easily readable and parsing (parse) is easy. The export AOF file is also very simple: for example, if you accidentally executed the Flushall command, but as long as the AOF file is not rewritten, just stop the server, remove the Flushall command at the end of the AOF file, and restart Redis, you can The data set reverts to the state before Flushall execution.

Disadvantage

• For the same dataset, the volume of the AOF file is usually larger than the size of the RDB file.

• Depending on the Fsync policy you are using, the AOF may be slower than the RDB. In general, the performance of Fsync per second is still very high, while closing fsync can make AOF as fast as an RDB, even under high load. However, the RDB can provide a more guaranteed maximum delay time (latency) when handling large write loads.

• AOF has been a bug in the past: Because of the reason of individual commands, the AOF file cannot be restored as it was when the data set was reloaded. (for example, a blocking command Brpoplpush has caused such a bug.) The test suite adds tests to this situation: they automatically generate random, complex datasets, and re-load the data to make sure everything is fine. Although this bug is not common in AOF files, it is almost impossible for an RDB to appear in comparison.

Choice

Please see the article at the beginning of the content, according to the needs of the business to make reasonable use

-----------------------------------------------------------------------------------------------

Redis persistence mechanism