Redis aof Preservation Mechanism

Online said AoF have three ways to save, do not automatically save, automatically save every second, each command automatically saved.

where each second automatically save this looks good, but may be a variety of IO time delay, so exactly how to judge the save, not too clear, so there is this article.

AOF Command Synchronization

Redis logs all commands (and their parameters) that have been written to the database to the AOF file to record the state of the database, which we call the recording process synchronous for convenience.

For example, if you execute the following command:

Redis>Rpush List1 2 3 4(integer)4Redis>Lrange List0 -11) "1"2) "2"3) "3"4) "4the Redis>KEYS*1) "List" Redis>rpop List "4the Redis>lpop List "1the Redis>Lpush List1(integer)3Redis>Lrange List0 -11) "1"2) "2"3) "3"

Then four of these write commands that have modifications to the database are synchronized to the AOF file:

1 2 3 4  1

For ease of handling, AOF files use the format of the network protocol to save these commands.

For example, the four commands listed above are actually saved in the AOF file as follows:

*2$6select$10*6$5rpush$4list$11$12$13$14*2$4rpop$4List*2$4lpop$4List*3$5lpush$4list$11

Except for the Select command, which is the AOF program itself, the other commands are the ones we executed in the terminal.

The entire process of synchronizing a command to a AOF file can be divided into three phases:

1. Command propagation: Redis sends information such as commands executed, command parameters, and the number of arguments to the command to the AOF program.
2. Cache Append: The AOF program converts the command into a network protocol format based on the received command data, and then appends the protocol content to the server's AOF cache.
3. File writes and saves: The contents of the AOF cache are written to the end of the AOF file, and if the set AOF save condition is met, the fsync function or fdatasync function is called, and the written content is actually saved to disk.

These three steps are described in detail in the following subsections.

Command propagation

When a Redis client needs to execute a command, it sends the protocol text to the Redis server over a network connection.

For example, to execute a command SET KEY VALUE , the client sends text to the server "*3\r\n$3\r\nSET\r\n$3\r\nKEY\r\n$5\r\nVALUE\r\n" .

After the server receives a request from the client, it chooses the appropriate command function based on the contents of the protocol text, and converts each parameter from the string literal to the Redis string object ( StringObject ).

For example, for the SET command examples above, Redis points the client's command pointer to the function that implements the SET command setCommand , and creates three Redis string objects, respectively, SET KEY and VALUE three parameters (the command counts as arguments).

Each time the command function executes successfully, the command arguments are propagated to the AOF program, and the REPLICATION program (this section does not discuss this, listed here for completeness only).

This process of executing and propagating commands can be represented by the following pseudocode:

If(Execrediscommand (cmdargvargc ) == exec_success): if aof_is_turn_on (): # propagate command to AOF program propagate_aof (cmdargvargc) if replication_is_turn_on (): # propagate command to REPLICATION program propagate_replication (cmd argvargc)        

The following is a flowchart of the process:

Cache Append

When the command is propagated to the AOF program, the program converts the command from the string object back to the original protocol text, based on the command and the parameters of the command.

For example, if the three parameters that the AOF program accepts, respectively SET , KEY and VALUE three strings are saved, it will generate the protocol text "*3\r\n$3\r\nSET\r\n$3\r\nKEY\r\n$5\r\nVALUE\r\n" .

After the protocol text is generated, it is appended to redis.h/redisServer the end of the structure aof_buf .

redisServerThe structure maintains the state of the Redis server, and the aof_buf domain holds all the protocol text waiting to be written to the AOF file:

{    //other fields ...    aof_buf//other domains ... };

At this point, the append command to the cache step is complete.

Together, the entire cache append process can be divided into the following three steps:

1. Information such as the command, the parameters of the command, the number of arguments, the database used, and so on.
2. Restore the command to the Redis network protocol.
3. Appends the protocol text to the aof_buf end.

File Write and save

Each time the server routine task function is executed, or the event handler is executed, the aof.c/flushAppendOnlyFile function is called, which performs the following two tasks:

Write: aof_buf writes the cache in to the AOF file according to the criteria.

Save: fsync saves the AOF file to disk, depending on the condition, call or fdatasync function.

Two steps need to be executed according to certain conditions, and these conditions are determined by the save mode used by AOF, the following subsections describe the three save modes used by AOF, and in these modes, the call conditions of the step WRITE and SAVE.

AOF Save mode

Redis currently supports three types of AOF save modes, namely:

1. AOF_FSYNC_NO: Do not save.
2. AOF_FSYNC_EVERYSEC: Save once every second.
3. AOF_FSYNC_ALWAYS: Save once for each command executed.

These three save modes are discussed in the following three subsections.

Do not save

In this mode, WRITE will be executed each time the function is called, flushAppendOnlyFile but SAVE will be skipped.

In this mode, SAVE is only executed in any of the following cases:

• Redis is turned off
• AOF function is turned off
• The system's write cache is flushed (either the cache is already full or the periodic save operation is executed)

The SAVE operation in these three cases will cause the Redis master process to block.

Save once every second

In this mode, save is executed in principle every second, because the save operation is called by the back-table thread, so it does not cause the server main process to block.

Note that the word "in principle" is used in the previous sentence, and in practice, the fsync call to or is not made once per second in this mode fdatasync , and it is related to the state in which Redis was called flushAppendOnlyFile when the function was invoked.

flushAppendOnlyFileThe following four scenarios may occur whenever a function is called:

• The child thread is executing SAVE, and:

  1. This save does not take more than 2 seconds to execute, then the program returns directly and does not perform WRITE or new save.
  2. This save has been executed for more than 2 seconds, then the program executes WRITE but does not perform a new save. Note that because write writes must wait for the enters upgradeable to finish (old) SAVE, write here will block longer than usual.

• The child thread is not executing SAVE, and:

  3. The last time the save was successfully executed is no more than 1 seconds, then the program executes WRITE but does not execute save.
  4. The last time the save has been successfully executed for more than 1 seconds, the program executes WRITE and save.

You can use a flowchart to indicate these four situations:

According to the above instructions, in the "save once per second" mode, if a failure occurs in condition 1, the user can lose up to less than 2 seconds of all data generated.

If a failure occurs in case 2, the data lost by the user can be more than 2 seconds.

What the Redis official says is that AOF is not accurate when it fails to "save once every second" and loses only 1 seconds of data.

Save once for each command executed

In this mode, both WRITE and SAVE are executed after each command is executed.

Also, because save is performed by the Redis master process, the main process is blocked during save execution and cannot accept command requests.

The impact of AOF save mode on performance and security

In the previous section, we briefly described how the three AOF save modes work, and now it's time to look at the differences in security and performance of these three patterns.

For the three AOF save modes, they block the server master process as follows:

1. Do not Save ( AOF_FSYNC_NO ): Writes and saves are performed by the main process, and two operations block the main process.
2. Save once per second ( AOF_FSYNC_EVERYSEC ): The write operation is performed by the main process, blocking the main process. The save operation is performed by a child thread and does not directly block the main process, but the speed at which the save operation completes affects the length of time the write operation is blocked.
3. Save once for each command ( AOF_FSYNC_ALWAYS ): As in mode 1.

Because blocking operations make it impossible for the Redis master process to process requests continuously, in general, the less the blocking operation executes and the faster it completes, the better the Redis performance.

The save operation of mode 1 only executes when AOF is closed or Redis is closed, or is triggered by the operating system, in general, this mode only needs to be blocked for write, so it writes more than the next two modes, of course, this performance is at the cost of reducing security: In this mode, if the operation The number of lost data is determined by the operating system's cache flushing strategy.

Mode 2 is better than pattern 3 in terms of performance, and in general, this mode loses no more than 2 seconds of data, so it is more secure than mode 1, which is a performance-and security-preserving scheme.

Mode 3 is the highest security, but the performance is also the worst because the server must block until the command information is written and saved to disk before the request can continue processing.

Combined, the operating characteristics of the three AOF modes can be summarized as follows:

Mode	is WRITE blocked?	is SAVE blocked?	amount of data lost during downtime
AOF_FSYNC_NO	Blocking	Blocking	The last time the operating system triggered the data after the SAVE operation on the AOF file.
AOF_FSYNC_EVERYSEC	Blocking	Do not block	In general, no more than 2 seconds of data.
AOF_FSYNC_ALWAYS	Blocking	Blocking	A maximum of one command's data is lost.

Redis aof Preservation Mechanism