Chapter 10 Redis persistence-RDB + AOF, redis-rdb

Chapter 10 Redis persistence-RDB + AOF, redis-rdb

Note: This article mainly references Redis Design and Implementation

1. Redis persistence Methods

• RDB
  • Execution mechanism: snapshot. The key-value binary format in databases is directly stored in the rdb file.
  • Advantage: High Performance (because it is a snapshot, and the execution frequency is lower than aof, And the binary form of key-values is directly stored in the rdb file, and data recovery is faster)
  • Disadvantage: if the data goes down between the save configuration conditions, the data will be lost.
• AOF
  • Execution mechanism: append each modification command to the aof file.
  • Advantage: data is not easy to lose
  • Disadvantage: low performance (each modification operation must be appended to the aof file. The execution frequency is higher than that of RDB, And the aof File Stores commands, for data recovery, You need to execute the command line by line, so the recovery is slow)

 

2. RDB

Actual configuration (in redis. conf)

# Any of the following three methods will write the cached content of the database to the rdb file (the write method is bgsave) # If you comment out the following three commands, you are not allowed to use rdbsave 900 1 #300 s. At least one key has changed. save 10000 10 #10000 S. At least 10 keys have changed. save 60 # There are at least keys after 60 s. key changed # When the rdb process in the background exports a snapshot (part of the key-value) when an error occurs in the rdb file process (that is, when the last backend storage fails ), # Whether the redis master process still accepts data writing to the database # This method will let the user know that an error occurred while the data is persisted to the hard disk (equivalent to a monitoring method ); # If a good redis persistent monitoring is installed, you can set it to "no" stop-writes-on-bgsave-error yes # Use LZF to compress the string, then write it to the rdb file # If you want the RDB process to save a little CPU time and set it to no, but the last rdb file may be large rdbcompression yes # After redis restarts, before writing data from the rdb file to the memory, check whether the rdb file is damaged (based on the check_sum In the rdb file) rdbchecksum yes # Set the rdb file name dbfilename dump. rdb # Set the rdb file storage directory dir./

Description:

• For more information about each configuration item, see the notes.

Note::

• For the save command, this command is configured and is executed in the background using bgsave.
  • Bgsave: The Redis master process reads and writes data. The RDB sub-process performs data persistence operations. During the persistence operation, the read and write operations of the master process are not blocked.
• If either of the preceding three save commands occurs, the bgsave command will occur, which has two problems. Assume that 10000 keys have changed (write, delete, update) within 60 s ), will the persistence be performed immediately? After this persistence, it is assumed that 240 s is passed, and no key change operation is performed during this period, does persistence occur at this time (because 10 keys are changed within 10000 S, keys are changed here )?
  • Persistence is not performed immediately: redis uses the serverCron function every MS by default to check whether the save configuration conditions are met. bgsave is performed if the conditions are met. In this way, if the conditions are within ms, I have already met the bgsave condition, so when I actually execute bgsave, I will wait until the serverCron is executed.
  • No persistence occurs: redis has two dirty parameters (record the number of key modifications after the last bgsave, and the above example is 0 in S) and lastsave (the last time when the bgsave command was successfully executed). The number of modifications to each save configuration in the configuration indicates dirty, and each time period is calculated starting from lastsave.
• Comment out all the save commands, RDB will not work
• Rdbcompression yes: is it required to perform a compression operation when every string is stored in the rdb file?
  • No: After the value is set to yes, the string is compressed only when the length of the string is greater than or equal to 21 bytes.
• Rdbchecksum yes: Where is the checksum stored? Why can I check whether the file is damaged by comparing the checksum?
  • Check_sum is stored in the last eight bytes of the RDB file (for details about the RDB file structure, refer to implementation of Redis "Chapter 1 RDB persistence "), the simple RDB file structure is as follows:
    • The first five bytes starting with the RDB file "REDIS" are the criteria for determining whether a file is an RDB file (similar to the "magic number" in the class file ")
    • The next four bytes: RDB file version (db_version)
    • Databases (plural): stores the key-value information stored in each database redisDb (the core of data persistence and restoration)
    • EOF (1 byte): End of the RDB file body
    • Check_sum (8 bytes): Check. This value is calculated based on the first and fourth scores. During persistence, this value is calculated and written to the end of the rdb file; when data is restored Based on the rdb file, a checksum is calculated based on the first and fourth scores in the rdb file, and then compared with the check_sum (the last eight bytes) in the current rdb file) the content of the file is compared. if the content is the same, the file is not damaged. if the content is different, the data in the first four parts is damaged (that is, the file is damaged)
• When the Redis server is started, redis automatically checks whether there are rdb files (if there is no aof). If yes, data is restored Based on the rdb file. Before the data recovery is complete, will block the client's read and write operations on redis

 

3. AOF

Actual configuration (in redis. conf)

# Whether to enable the aof log function (appendonly yes) appendonly no # aof file storage path and file name # appendfilename appendonly. aof # Every command is synchronized to the aof file immediately (safe, but slow, because every command performs a disk operation) # appendfsync always # write data once per second to the aof file appendfsync everysec # hand over the write work to the operating system. The operating system determines the buffer size and writes the data to the aof file in a unified manner (fast, but the synchronization frequency is low and data is easy to lose) # appendfsync no # Whether the aof operation stops when RDB persists data. If yes, the aof operation stops. # during this period of time, the executed commands will be written to the memory queue. After RDB persistence is complete, these commands will be uniformly written to the aof file # the configuration of this parameter takes into account the low frequency of RDB persistent execution, however, the execution time is long, while the AOF execution frequency is high and the execution time is short. # If two sub-processes (RDB sub-process and AOF sub-process) are executed simultaneously) low Efficiency (both sub-processes are disk read/write) # However, changing to yes may result in a long RDB persistent execution time, during this period, many commands are written to the memory queue. # In the end, the queue cannot be put down. In this way, the AOF command written to the AOF file may be much less # when data is restored, recovery from the aof file will result in a lot of data loss # so you can choose no. no-appendfsync-on-rewrite no # AOF rewrite: Convert the data in the memory into commands, then write these commands into the aof file # rewrite purpose: assume that we perform 100 operations on the same key in the memory, and the value of the key is 100, # There will be 100 command logs in aof. In this case, there are two disadvantages: #1) The AOF file is too large to occupy hard disk space 2) restoring data based on the AOF file is extremely slow (100 commands need to be executed) # If we convert the key in the memory to "set key 100" and then write it to the aof file, # The size of the aof file will be greatly reduced, and data will be restored quickly based on the aof file (only one command is required) # Note: if both constraints are met, aof rewrite occurs. # If there is no second one, you only need to add some data in the early stage of aof, aof rewriting occurs # When the percentage of aof growth is 100% (that is, twice the original size, for example, 100 m, the next rewrite is when the aof file is MB ), AOF rewrite auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64 mb # AOF rewrite only occurs when the aof file is larger than 64 mb

Note:

• For more information about each configuration item, see the notes.

Note:

• Each client command is directly written to the AOF buffer when executed.
  • Appendfsync always: after each command enters the buffer zone, it is immediately appended to the AOF file from the buffer zone.
  • Appendfsync everysec: append all commands in the buffer to the AOF file every second.
  • Appendfsync no: after each command enters the buffer zone, the operating system determines when (when the buffer zone is full) to append all the commands in the buffer zone to the AOF file.
• Aof rewrite must meet two conditions in the configuration file
• Aof rewrite adopts the latter sub-process for execution
  • Aof rewrite will perform a lot of write operations. If you use a single thread to do this, the main process will be blocked for a long time (redis is a single thread), and the client's read and write will become invalid at this time.
  • Problems caused by using aof background rewriting process:
    • If a new command is used to read and write the database during the background process rewriting, the new aof file is different from the storage content of the database. (Note: During the background process rewriting, the read/write operations on the database will also enter the aof buffer, or the aof FILE command will be executed to write, but note that, although all the commands in this period are still written to the aof file, the aof file will be replaced by the new aof file after the rewrite. The new aof file does not have these commands, if a crash occurs before the next rewrite and the aof is used to restore the database, a lot of commands will be lost)
    • Solution: Set an aof rewrite buffer, which is only used to write the database read/write commands that occur to the rewrite buffer during background process rewriting (of course, in this case, the server process not only writes the database read/write commands to the rewrite buffer, but also writes them to the aof buffer to ensure normal aof operations.) After the rewrite sub-process completes the rewrite, send a signal to the master process of the server. At this time, the master process appends the commands in the aof rewrite buffer to the new aof file and replaces the old aof file with the new aof file.
  • Note:
    • During background process rewriting, when writing database read/write commands to the rewrite buffer, why do these commands need to be written to the aof buffer (because these commands will be written to the old aof file )?
      
      • If the old aof file can be replaced by a new aof file in the final master process (that is, it will not go down before this ), writing data to the aof buffer is useless (because it will be overwritten). However, if the data goes down before this, our aof file is still the old aof file, at this time, the command is written to the aof buffer to ensure that the aof file is saved as many commands as possible. In the future, it will be used to restore the most lost data in the database, that is, 1 s. (Appendfsync everysec)
    • During this operation, if a new client read/write command is available, will be blocked (because the master process is performing the above operations ). This is the only blocked part of the aof rewrite process.

Summary:

• If both RDB and AOF are configured, data persistence is performed. However, when data is recovered based on the file, the RDB file is invalidated based on the AOF file.
  • Note: Data Recovery is a blocking operation (any client read/write requests that come here are invalid)
• Bgsave and bgrewriteaof (background aof rewrite) commands cannot occur simultaneously
  • If bgsave is being executed, bgrewriteaof will be executed after bgsave is executed.
  • If bgrewriteaof is being executed, bgsave is discarded.
• RDB and AOF can be configured at the same time, but the aof file is used to restore the database.