Redis source code analysis 18-persistence aof

The purpose of Redis's aof function is to provide better support for the persistence mechanism in terms of performance and persistence granularity. Snapshot persistence has two granularities: time (in seconds) and the number of changed keys. If the persistence granularity is small, the performance will be greatly affected, because each time the entire db is dumped; if the persistence granularity is large, the number of objects is specified within the specified time.

The purpose of Redis's aof function is to provide better support for the persistence mechanism in terms of performance and persistence granularity. Snapshot persistence has two granularities: time (in seconds) and the number of changed keys. If the persistence granularity is small, the performance will be greatly affected, because each time the entire db is dumped; if the persistence granularity is large, the number of objects is specified within the specified time.

The purpose of Redis's aof function is to provide better support for the persistence mechanism in terms of performance and persistence granularity.

Snapshot persistence has two granularities: time (in seconds) and the number of changed keys. If the persistence granularity is small, the performance will be greatly affected, because each time the entire db is dumped; if the persistence granularity is large, the persistence of a specified number of data within the specified time cannot be guaranteed. The aof persistence granularity is the command that modifies the database data each time. Therefore, the granularity is the smallest, which is similar to the log method. Because only one command is recorded, the performance is also the best. In addition, similar to logs, if the aof file is larger and larger, you can run the BGREWRITEAOF command to recreate the file in the background.

Let's take a look at how redis records commands.

The call function is a function for command execution (this function has been described in detail in the previous command processing section ). If data is modified before and after the command is executed, the value of server. dirty varies. When the aof mechanism is enabled, the call function calls feedAppendOnlyFile to save the command and its related parameters.

static void call(redisClient *c, struct redisCommand *cmd){   long long dirty;   dirty = server.dirty;   cmd->proc(c);   dirty = server.dirty-dirty;   if(server.appendonly && dirty)       feedAppendOnlyFile(cmd,c->db->id,c->argv,c->argc);   ---}

FeedAppendOnlyFile first checks whether the db where the current command is located is consistent with the db where the previous command is executed. If they are inconsistent, you need to release a select command for selecting the database and then perform some command conversion (the code is omitted ).

Then, save the buf corresponding to the command parameters to the server. in aofbuf, this parameter stores the commands executed by redis and their parameters for a period of time. redis will fl them to the aof file on the disk at an appropriate time; if the aof file is rebuilt in the background, the buffer is also saved to the server. in bgrewritebuf, the buffer zone stores the commands executed by redis and their parameters when the background process of the reconstruction aof file is running. When the background process exits, these commands must be saved to the Reconstruction file.

static void feedAppendOnlyFile(struct redisCommand *cmd, int dictid, robj **argv, int argc){   ---   server.aofbuf = sdscatlen(server.aofbuf,buf,sdslen(buf));   ---   if(server.bgrewritechildpid != -1)       server.bgrewritebuf = sdscatlen(server.bgrewritebuf,buf,sdslen(buf));   sdsfree(buf);}

Let's see when server. aofbuf will be refreshed to the disk aof file.

The flushAppendOnlyFile function is used for Refresh. This function will be called in beforeSleep (this function has been introduced in the event processing section ), this function is executed before processing client events (the event loop function aeMain executes beforesleep first and then aeProcessEvents). Therefore, server. the value in aofbuf will be refreshed to the disk before sending a response to the client.

FlushAppendOnlyFile calls write to write all servers at a time. the data in the aofbuf buffer is synchronized by calling aof_fsync Based on the configured synchronization policy, in this way, the new commands and their parameters are appended to the aof file.

static void flushAppendOnlyFile(void){   time_t now;   ssize_t nwritten;   ---    nwritten = write(server.appendfd,server.aofbuf,sdslen(server.aofbuf));   ---   sdsfree(server.aofbuf);   server.aofbuf = sdsempty();   /* Fsync if needed */   now = time(NULL);   if(server.appendfsync == APPENDFSYNC_ALWAYS||        (server.appendfsync == APPENDFSYNC_EVERYSEC &&        now-server.lastfsync > 1))   {       /* aof_fsync is defined as fdatasync() for Linux in order to avoid         * flushing metadata. */       aof_fsync(server.appendfd);/* Let's try to get this data on the disk */       server.lastfsync = now;   }}

Next, let's take a look at how to recreate the aof file in the background.

Aof reconstruction is completed by calling the rewriteAppendOnlyFileBackground function. You can view the call relationship of the function. The function will be executed after the bgrewriteaof command is received, and will be called when the config command is received and the aof mechanism is never used to enable the aof mechanism, it will also be called when the system running redis acts as the slave, establishes a connection with the master, and runs syncWithMaster in the serverCron function.
RewriteAppendOnlyFileBackground ):

1) use fork to create a sub-process

2) The sub-process calls rewriteAppendOnlyFile to write data and commands that reflect the current db status in a temporary file,

At this time, the parent process will put the commands executed during this period of time that can change the current db data into server. bgrewritebuf (see the preceding description of feedAppendOnlyFile)

3) when the child process exits, the parent process receives a signal to flush the data in the above memory buffer to the temporary file, and then rename the temporary file into a new aof file (backgroundRewriteDoneHandler ).

The parent process is a child process in the serverCron function waiting for aof rewriting or snapshot storage. The Code is as follows:

/* Check if a background saving or AOF rewrite in progress terminated */  if(server.bgsavechildpid != -1||server.bgrewritechildpid != -1){      int statloc;      pid_t pid;      if((pid = wait3(&statloc,WNOHANG,NULL))!= 0){          if(pid == server.bgsavechildpid){              backgroundSaveDoneHandler(statloc);          } else {              backgroundRewriteDoneHandler(statloc);          }          updateDictResizePolicy();      }  }

RewriteAppendOnlyFile writes the commands and parameters that reflect the current db status to a temporary file. This function traverses every piece of data in the db. The db in redis is actually a large hash table, and each piece of data is represented by (key, val. You can know the val type from the key (redis supports REDIS_STRING, REDIS_LIST, REDIS_SET, REDIS_ZSET, and REDIS_HASH), and then decode the data in val. Write data in the form of commands executed by the client. For example, for the REDIS_STRING type, write "* 3 \ r \ n $3 \ r \ nSET \ r \ n", then write the set key, and then write the val; for REDIS_LIST type, after val is forcibly converted to list type, write "* 3 \ r \ n $5 \ r \ nRPUSH \ r \ n" first ", write the name of the list to be operated, write the first data of the list, and cycle the first three steps until the list traversal is complete. For the REDIS_SET type, write "* 3 \ r \ n $4 \ r \ nSADD \ r \ n" for each data entry. For the REDIS_ZSET type, write "* 4 \ r \ n $4 \ r \ nZADD \ r \ n" for each data entry. For the REDIS_HASH type, write "* 4 \ r \ n $4 \ r \ nHSET \ r \ n" for each piece of data (simple but trivial code, omitted ).

Finally, we will introduce how to use aof to reconstruct the database when redis is started.

The key to rebuilding upon startup is to build a fake client and then use this client to send the commands read from the aof file to the server.

Int loadAppendOnlyFile (char * filename) {--- fakeClient = createFakeClient (); while (1) {--- if (fgets (buf, sizeof (buf), fp) = NULL) {---} // parse the buf as the corresponding command and parameter // SEARCH Command cmd = lookupCommand (argv [0]-> ptr ); --- // Execute Command cmd-> proc (fakeClient );---}---}

Original article address: redis source code analysis 18-persistence aof, thanks to the original author for sharing.