Redis's aof persistence in-depth parsing

Redis offers two persistence modes, RDB and aof; unlike an RDB, aof can completely record an entire database, unlike an RDB that is just a snapshot of a database at a time;

So why does the aof pattern completely record the entire database?

Principle: In aof mode, Redis will record each update command executed and save it to the AoF file, and when Redis needs to recover the database data, it only needs to read the command sequentially from the previously saved aof file to execute the eg.

Shell code we executed the following command: Redis 127.0.0.1:6379> set name Diaocow OK redis 127.0.0.1:6379> Lpush country China USA (integer) 4 The class capacity in the AoF file is similar to the following: *3\r\n$3\r\nset\r\n$4\r\nname\r\n$7\r\ndiaocow\r\n *4\r\n$5\r\nlpush\r\n$7\r\ncou ntry\r\n$5\r\nchina\r\n$3\r\nusa\r\n
Looking at the above, I think I don't have to explain too much, you can also roughly guess the AOF protocol format, because it is too simple and clear

Redis logs the update command to the AoF file, which is divided into two stages:

Phase 1: Write update command to AOF cache

Python code    def processcommand (CMD, ARGC, ARGV):       #   Execute Commands        call (CMD, ARGC, ARGV)         #  This command changes the key space and aof mode opens        if redisServer.update_key_space  and redisserver.aof_state & redis_aof_on:            feedappendonlyfile (CMD, ARGC, ARGV)        def  Feedappendonlyfile (CMD, ARGC, ARGV):       #  Convert command to AOF protocol format         AOFCMDSTR = GETAOFPROTOCOLSTR (CMD, ARGC, ARGV)        redisserver.aof_buf.append (aofcmdstr )            #  there is a subprocess in progress Aof_rewrite (about Aof_rewrite, say later)        if  redisserver.aof_child_pid != -1:           redisserver.aof_rewrite_buf_blocks.append (aofcmdstr )   
Phase 2: Write aof cache to file

When we start the next event loop, Redis writes the contents of the AOF cache to the file:
Python Code def flushappendonlyfile (force): If Len (redisserver.aof_buf) = = 0:return # Write cache data to file If Writebypolicy (Force, Redisserver.aof_fsync): Write (REDISSERVER.AOF_FD, Redisserver.aof_buf, Len (redi SSERVER.AOF_BUF) # Sync data to Hard disk if fsyncbypolicy (Force, Redisserver.aof_fsync): Fsync (redisserver . aof_fd)
For more details see: aof.c/flushappendonlyfile function (PS: This function code looks rather obscure)

See here, you may have two questions:
1. Why do I call the Fsync function instead of writing the data to the file by calling write?
2. What is aof_fsync in pseudo-code and it has several types.

First answer question 1, why write the file, also call the Fsync function:
Most Unix systems use "deferred write" technology in order to reduce disk IO, which means that when we finish the write call, the data is not necessarily written to disk immediately (probably still in the system's buffer cache or page cache), so that when the host suddenly loses power, The data that we thought had been written to the disk file might be lost, so when we need to make sure that the data is fully and correctly written to the disk (such as the persistence of the database), you need to call the synchronization function Fsync, which will block until the data is all written to the hard disk

What is the problem 2,AOF_FYSNC:
Aof_fsync is used to specify the flush policy, which is the policy to invoke the Fsync function, which has three types:
A. Aof_fsync_no: Each time the contents of the Aof_buf are written to disk, but the FSYNC function is not called;
B. Aof_fsync_always: Each time the contents of the Aof_buf are written to disk, and the FSYNC function is called;
C. aof_fsync_everysec

Since Aof_fsync_always will call FSYNC every time a file is written, this flush policy guarantees the integrity of the data, with the disadvantage of poor performance (because FYSNC is a synchronous call that blocks the processing of the client request by the main process), and AOF_FSYNC_ No due to the reliance on the operating system automatic sync, it can not guarantee the integrity of the data;

There is no way to compromise: not to reduce the performance of the system, but also to maximize the integrity of the data, the answer is: aof_fsync_everysec,aof_fsync_everysec flush strategy is: regular (at least 1s) to call FSYNC , and the operation is placed in an asynchronous queue (thread) to execute, so it does not block the main process

AOF the way the background executes and the RDB has a similar place, fork a child process, the main process is still serving, the child process executes AOF persisted, the data is dump to disk. Unlike an RDB, the master process records all data changes during the period (the main process is still in service) and is stored in the server.aof_rewrite_buf_blocks, and the Redis update cache is appended to the AoF file after the end of the stage process, which is Not available for RDB persistence.

The

AoF mode is basically complete, but as Redis runs, the aof file becomes larger (faster in business high-staging) for two: 
A. The AOF protocol itself is a text protocol, and the comparison takes up space; 
B. Redis needs to record all update commands from start to current; 

These two reasons cause the AoF file to become very large, and how can you optimize it? For example, a user executes three commands: Lpush name Diaocow; Lpush name Jack; The Lpush name jobs 
Aof file records the following data: 
Aof_file code    *3\r\n$5\r\nlpush\r\n$4\r\nname\r\n$7\r\ ndiaocow   *3\r\n$5\r\nlpush\r\n$4\r\nname\r\n$4\r\njack   *3\r\n$5\r\nlpush\r\n$4\r\nname\r \n$4\r\njobs  
But in fact only need to record one: Lpush name diaocow Jack Jbos command can: 
Aof_file code    *5\r\n$5\r \nlpush\r\n$4\r\nname\r\n$7\r\ndiaocow\r\n$4\r\njack\r\n$4\r\njobs  
So when the AOF file reaches REDIS_AOF_REWRITE_ Min_size (1M), Redis executes aof_rewrite to optimize aof files;