AOF--Redis design and implementation

Redis provides two persistence mechanisms for RDB and AOF: The RDB saves a snapshot of the database (snapshot) to disk in a binary way. AOF, in the form of protocol text, logs all commands (and their parameters) that have been written to the database to the AOF file to achieve the purpose of logging the database state.

This chapter first describes how the AOF function works, how the commands are saved to the AOF file, and how the different AOF save modes affect data security and Redis performance.

It then describes how to recover the state of the database from the AOF file, and the implementation mechanism behind the method.

Finally, we will introduce the method of rewriting AOF to adjust the volume of the file, and study how this method is carried out without changing the state of the database.

Since this chapter deals with the mechanism of AOF operation, if you have not yet understood the AOF functionality, please read the section on AOF in the Redis persistence manual.

AOF Command Synchronization

Redis logs all commands (and their parameters) that have been written to the database to the AOF file to record the state of the database, which we call the recording process synchronous for convenience.

For example, if you execute the following command:

redis> Rpush List 1 2 3 4
(integer) 4

redis> lrange list 0-1
1) "1"
2) "2"
3) "3"
4) "4" 
  redis> KEYS *
1) "list"

redis> rpop list
"4"

redis> lpop list
"1"

redis> Lpush list 1
(integer) 3

redis> lrange list 0-1
1) "1"
2) "2"
3) "3"

Then four of these write commands that have modifications to the database are synchronized to the AOF file:

Rpush List 1 2 3 4

rpop list

lpop list

lpush list 1

For ease of handling, AOF files use the format of the network protocol to save these commands.

For example, the four commands listed above are actually saved in the AOF file as follows:

* *
$6
SELECT $
0 *6 $
rpush
$4 List $1 $
2
$
3 $4 *
$4
rpop
$4
list
*
$4
lpop
$4
list
lpush
$4 list $
1

Except for the SELECT command, which is the AOF program itself, the other commands are the ones we executed in the terminal.

The entire process of synchronizing a command to a AOF file can be divided into three stages: command propagation: Redis sends information such as commands executed, command parameters, and the number of arguments to the command to the AOF program. Cache Append: The AOF program converts the command into a network protocol format based on the received command data, and then appends the protocol content to the server's AOF cache. File writes and saves: The contents of the AOF cache are written to the end of the AOF file, and if the set AOF save condition is met, the Fsync function or Fdatasync function is called, and the written content is actually saved to disk.

These three steps are described in detail in the following subsections. Command Propagation

When a Redis client needs to execute a command, it sends the protocol text to the Redis server over a network connection.

For example, to execute the command SET KEY VALUE, the client sends the text "*3\r\n$3\r\nset\r\n$3\r\nkey\r\n$5\r\nvalue\r\n" to the server.

After the server receives a request from the client, it chooses the appropriate command function based on the contents of the protocol text, and converts each parameter from the string literal to the Redis string object (Stringobject).

For example, for the SET command examples above, Redis points the client's command pointer to the SetCommand function that implements the set command, and creates three Redis string objects that hold SET, KEY, and VALUE three parameters (the command counts as arguments).

Each time the command function executes successfully, the command arguments are propagated to the AOF program, and the REPLICATION program (this section does not discuss this, listed here for completeness only).

This process of executing and propagating commands can be represented by the following pseudocode:

if (Execrediscommand (cmd, argv, argc) = = exec_success):

    if aof_is_turn_on ():
        # Propagate command to AOF program
        propagate_aof ( CMD, argv, argc)

    if replication_is_turn_on ():
        # Propagate command to Replication program
        propagate_replication (cmd, argv, argc )

The following is a flowchart of the process:

Cache Append

When the command is propagated to the AOF program, the program converts the command from the string object back to the original protocol text, based on the command and the parameters of the command.

For example, if the AOF program accepts three parameters that hold the SET, KEY, and VALUE three strings, it will generate the protocol text "*3\r\n$3\r\nset\r\n$3\r\nkey\r\n$5\r\nvalue\r\n".

After the protocol text is generated, it is appended to the end of the aof_buf of the redis.h/redisserver structure.

The REDISSERVER structure maintains the status of the Redis server, and the AOF_BUF domain holds all the protocol text waiting to be written to the AoF file:

struct Redisserver {

    //other domain ...

    SDS Aof_buf;

    Other domains ...
};

At this point, the append command to the cache step is complete.

Together, the entire cache append process can be divided into the following three steps: Accept the command, the parameters of the command, and the number of parameters, the database used and other information. Restore the command to the Redis network protocol. Appends the protocol text to the end of the aof_buf. file Write and save

Each time the server routine task function is executed, or the event handler is executed, the Aof.c/flushappendonlyfile function is called, which performs the following two tasks:

Write: Writes the cache in Aof_buf to the AoF file, depending on the condition.

Save: Depending on the condition, call the Fsync or Fdatasync function to save the AOF file to disk.

Two steps need to be executed according to certain conditions, and these conditions are determined by the save mode used by AOF, the following subsections describe the three save modes used by AOF, and in these modes, the call conditions of the step WRITE and SAVE. AOF Save Mode

Redis currently supports three AOF save modes, namely: Aof_fsync_no: Not saved. Aof_fsync_everysec: Save once every second. Aof_fsync_always: Save once for each command executed.

These three save modes are discussed in the following three subsections. Do not save

In this mode, WRITE will be executed each time the Flushappendonlyfile function is called, but SAVE will be skipped.

In this mode, SAVE will only be executed in any of the following cases: Redis is turned off AOF function is turned off the write cache of the system is flushed (either the cache is already full, or the periodic save operation is executed)

The SAVE operation in these three cases will cause the Redis master process to block. Save once every second

In this mode, save is executed in principle every second, because the save operation is called by the back-table thread, so it does not cause the server main process to block.

Note that the word "in principle" is used in the previous sentence, and in practice, the program's call to Fsync or Fdatasync in this mode is not once per second, as it relates to the state of Redis at the time the Flushappendonlyfile function is called.

Each time the Flushappendonlyfile function is called, the following four scenarios may occur:

The child thread is executing save and: The save has not been executed for more than 2 seconds, then the program returns directly and does not perform WRITE or new save. This save has been executed for more than 2 seconds, then the program executes WRITE but does not perform a new save. Note that because write writes must wait for the enters upgradeable to finish (old) SAVE, write here will block longer than usual.

The child thread is not executing save, and: The last successful execution of the save is no more than 1 seconds, then the program executes WRITE but does not execute save. The last time the save has been successfully executed for more than 1 seconds, the program executes WRITE and save.

You can use a flowchart to indicate these four situations:

According to the above instructions, in the "save once per second" mode, if a failure occurs in condition 1, the user can lose up to less than 2 seconds of all data generated.

If a failure occurs in case 2, the data lost by the user can be more than 2 seconds.

What the Redis official says is that AOF is not accurate when it fails to "save once every second" and loses only 1 seconds of data. save once for each command executed

In this mode, both WRITE and SAVE are executed after each command is executed.

Also, because save is performed by the Redis master process, the main process is blocked during save execution and cannot accept command requests. The impact of AOF Save mode on performance and security

In the previous section, we briefly described how the three AOF save modes work, and now it's time to look at the differences in security and performance of these three patterns.

For the three AOF save modes, they block the server master process as follows: Do not save (AOF_FSYNC_NO): Both write and save are performed by the main process, and two actions block the main process. Save once per second (AOF_FSYNC_EVERYSEC): The write operation is performed by the main process, blocking the main process. The save operation is performed by a child thread and does not directly block the main process, but the speed at which the save operation completes affects the length of time the write operation is blocked. Save once for each command (aof_fsync_always): Same as Mode 1.

Because blocking operations make it impossible for the Redis master process to process requests continuously, in general, the less the blocking operation executes and the faster it completes, the better the Redis performance.

The save operation of mode 1 only executes when AOF is closed or Redis is closed, or is triggered by the operating system, in general, this mode only needs to be blocked for write, so it writes more than the next two modes, of course, this performance is at the cost of reducing security: In this mode, if the operation The number of lost data is determined by the operating system's cache flushing strategy.

Mode 2 is better than pattern 3 in terms of performance, and in general, this mode loses no more than 2 seconds of data, so it is more secure than mode 1, which is a performance-and security-preserving scheme.

Mode 3 is the highest security, but the performance is also the worst because the server must block until the command information is written and saved to disk before the request can continue processing.

Combined, the operating characteristics of the three AOF modes can be summarized as follows:

Mode	WRITE is blocked.	whether SAVE is blocked.	amount of data lost during downtime
Aof_fsync_no	Blocking	Blocking	The last time the operating system triggered the data after the SAVE operation on the AOF file.
Aof_fsync_everysec	Blocking	Do not block	In general, no more than 2 seconds of data.
Aof_fsync_always	Blocking	Blocking	A maximum of one command's data is lost.

Read and data restore of AOF files

The AOF file holds the Redis database state, and the files contain command text that conforms to the REDIS protocol format.

This means that the database state of Redis can be restored as long as all the commands indicated in the AOF file are re-executed according to the protocol in the.

The detailed steps for Redis to read the AOF file and restore the database are as follows: Create a pseudo-client with no network connection (fake). Reads the text saved by the AOF and restores the command, the parameters of the command, and the number of commands based on the content. Executes the command using a pseudo-client, depending on the command, the parameters of the command, and the number of commands. Execute 2 and 3 until all the commands in the AOF file have been executed.

After completing the 4th step, the database saved by the AOF file will be completely restored.

Note that because the Redis command can only be executed in the context of the client, and the command used by the AOF restore comes from the AOF file instead of the network, the program uses a pseudo-client that does not have a network connection to execute the command. The effect of a pseudo-client executing a command is exactly the same as executing a command with a network-attached client.

The entire read and restore process can be represented by the following pseudo-code:

Def read_and_load_aof ():

    # Opens and reads AOF file
    = open (aof_file_name) while
    file.is_not_reach_eof ():

        # A Redis command that reads in a protocol text format
        cmd_in_text = File.read_next_command_in_protocol_format ()

        # finds command functions based on text commands, and create object cmd such as parameters and number of arguments
        , argv, argc = Text_to_command (cmd_in_text)

        # Execute command
        execrediscommand (cmd, argv, ARGC)

    # Close File
    file.close ()

As an example, here is a brief description of the AOF file:

* * $6 SELECT $0 for SET $ $ $ $ $
*8
$
Rpush
$4
list
$
1 $2 $3 $
4 $
5
6

When the program reads into the AOF file, it executes the Select 0 command first-the SELECT command is generated automatically by the AOF writer, which ensures that the program can restore the data to the correct database.

Then execute the following SET key value and Rpush 1 2 3 4 command to restore the key and list two key data.

To avoid having an impact on the integrity of the data, only the database-independent subscription and publishing functions can be used when the server loads the data, and all other commands return errors. AOF rewrite