Tenth Redis Persistence--rdb+aof

Note: This article mainly refers to the design and implementation of Redis

1. Two persistent modes of Redis

• Rdb
  • Execution mechanism: snapshot, storing the binary form of key-value in databases directly in the Rdb file
  • Advantage: High performance (because it is a snapshot, and the execution frequency is lower than aof, and the Rdb file is stored directly in the binary form of the key-values, also fast for recovering data)
  • Disadvantage: If there is a downtime between the save configuration conditions, the data will be lost
• AOF
  • Execution mechanism: appends each modification command to the data to the AoF file
  • Pros: Data is not easy to lose
  • Disadvantage: Low performance (each modification operation is appended to the AoF file, the execution frequency is higher than the RDB, and the aof file is stored in the command, for the recovery of data requires line-by-row execution command, so recovery slow)

2. RDB

The configuration used in practice (in redis.conf)

#发生以下三种的Any one writes the cached contents of the database to the Rdb file (written in bgsave) #若将下述的三条命令都注释掉, Rdbsave 1 # is not allowed      At least one key changed after 900s save      #300s后至少有10个key发生了变化save 10000    #60s后至少有10000个key发生了变化 # When the background RDB process exports a snapshot (part of the Key-value) to an RDB file (that is, when the last background save failed), #redis主进程是否还接受向数据库写数据 # This way will let the user know that an error occurred while persisting data to the hard disk (  Equivalent to a monitoring ), #如果安装了很好的redis持久化监控, can be set to "no" stop-writes-on-bgsave-error yes# use Lzf compressed string, and then write to the Rdb file # If you want the RDB process to save a bit of CPU time, set to No, but maybe the last Rdb file will be very large rdbcompression yes# whether the Rdb file is corrupted before it is written to memory from the Rdb file after a Redis reboot ( Depending on the checksum check_sum in the Rdb file Rdbchecksum yes# set the Rdb file name Dbfilename dump.rdb# set the storage directory for the Rdb files dir./

Description :

• See notes for detailed descriptions of each configuration

Note :

• For the Save command, the command is configured, and the background is performed in Bgsave.
  • Bgsave:redis The main process for data read and write operations, the RDB sub-process for data persistence operation, in the persistence operation, does not block the main process read and write operations
• If any of the above three save commands occur, the Bgsave command will occur, and there are two problems, assuming that 10,000 keys in 60s have changed (write, delete, update), will it be persisted immediately? After this persistence, assume that there is another 240s, and there is no change in the key operation, at this time if there is a persistence (because the 10 key changes to meet 300s, this is changed 10,000 key)?
  • does not immediately persist : Redis defaults every 100ms using the Servercron function to check if the condition of the Save configuration is met , and the bgsave is satisfied, so that if within 100ms, I have fulfilled the conditions of bgsave, so when I actually execute bgsave, I have to wait until Servercron executes.
  • No more persistence: Redis has two parameters dirty (record the number of changes to key after the last Bgsave, the top of the example in 240s is 0), and Lastsave (the last time the Bgsave command was executed successfully), in the configuration The number of modifications per save configuration refers to Dirty, and each time period is calculated as the starting point for Lastsave .
• Commenting out all the save commands, the RDB will not work
• Rdbcompression YES: configured so that every string is compressed once it is stored in an RDB file?
  • Not: After set to Yes, compression occurs only if the length of the string is greater than or equal to 21 bytes
• Rdbchecksum Yes: Where is this checksum stored? Why is it possible to determine if a file is damaged by a comparison checksum?
  • The checksum (check_sum) is stored in the last eight bytes of the Rdb file (the detailed RDB file structure, see "Redis This based on implementation", "chapter 10th RDB Persistence"), the simple RDB file structure is as follows:
    • The first five bytes of "REDIS" at the beginning of an RDB file are the criteria for judging whether a file is an Rdb file (similar to the "magic number" in a class file)
    • Next 4 bytes: Rdb file version number (db_version)
    • Databases (note is plural ): This stores key-value information stored in each library Redisdb (the core of the entire data persistence and recovery)
    • EOF (1 bytes): End of Rdb file body
    • Check_sum (8 bytes): Test and, the value is calculated according to the front four points, the value is calculated at the time of persistence and written to the end of the Rdb file, when the data is recovered according to the Rdb file, a checksum is computed based on the four points in the Rdb file. Then match the contents of the check_sum (that is, the last eight bytes) in the current Rdb file, if the same, the description is not corrupted, if not the same, the first four parts have data corruption (that is, the file corruption)
• When the Redis server is started, Redis automatically detects if there is an RDB file (provided that there is no aof) and, if so, recovers the data based on the Rdb file, which blocks the client's read and write operations to Redis until the recovery data is complete

3, AOF

The configuration used in practice (in redis.conf)

# whether to open the AoF log function (appendonly Yes) appendonly no# aof file's storage path and file name # appendfilename appendonly.aof# every command, immediately sync to the AoF file (very safe, But the speed is slow, because each command will do a disk operation) # Appendfsync always# writes data once to aof files per second Appendfsynceverysec#将写入工作交给操作系统, by the operating system to determine the size of the buffer, unified write to the AoF file (fast, but low synchronization frequency, easy to lose data) # Appendfsync no# when the Rdb persisted data, at this time the aof operation is stopped, if yes stop # During the time that is stopped, the executed commands are written to the memory queue, and when the RDB is persisted, the commands are written to the aof file uniformly. # This parameter is configured to take into account the low frequency of the RDB persistence execution, but the long execution time and the high frequency and short execution time of the aof execution, # If two sub-processes (Rdb subprocess, aof subprocess) are executed at the same time (two sub-processes are disk read-write) # But if you change to yes the result is that because the RDB persists for a long time, there are a lot of commands written to the memory queue during that time, # finally causing the queue to not fit, This way, the commands written to the AoF file may be a lot less aof. # When recovering data, you will lose a lot of data based on the aof file recovery # So, choose No good no-appendfsync-on-rewrite no# aof rewrite: Invert in-memory data into commands, Then write these commands back to the aof file # Rewrite purpose: Assuming that we have 100 operations on the same key in memory, and finally the value of the key is 100,# then there will be 100 command logs in aof, so there are two disadvantages: # 1)aof file is too large, Occupy hard disk space 2)very slow recovery of data based on AOF files(Need to execute 100 commands) # If we invert the key in memory into "set key 100" and then write to the AoF file, # Then the size of the AoF file will be greatly reduced, and the data recovered according to the aof file is very fast (only 1 commands need to be executed) # Note: Two constraints belowAll to meetThe AOF rewrite occurs only if there is no second, then in the early aof, as long as slightly add some data, it occurs aof rewrite # when the aof growth percentage is the original 100% (that is, the original size of twice times, for example, the original is 100m, The next rewrite is when the aof file is 200m), aof rewrite auto-aof-rewrite-percentage auto-aof-rewrite-min-size 64mb #AOF重写仅发生在当aof文件大于64m时

Description

• See notes for detailed descriptions of each configuration

Attention:

• Each client command writes commands directly to the AOF buffer at execution time
  • Appendfsync always: Once each command enters the buffer, it is immediately appended to the AoF file from the buffer
  • appendfsync everysec: appends all the commands in the buffer to the AoF file after each second
  • Appendfsync No: After each command enters the buffer, it is up to the operating system to determine when (mostly the buffer is almost full) to append all the commands in the buffer to the AoF file
• AoF rewrite needs to meet two conditions in the configuration file
• AoF rewrite using post-stage process execution
  • AoF rewrite will be a lot of write operations, if the use of single-threaded to do this, will be a long time to block the main process (Redis is a single thread), this time the client read and write will be invalidated
  • using the aof background rewrite process:
    • If there are new commands to read and write to the database during background process rewriting, the new aof file is different from the storage content of the database. (note: During a background process rewrite, read and write operations to the database will also go into the aof buffer, or execute the command write of the aof file, but note that although all the commands in this period are still written to the AoF file, the aof file is replaced with the new AOF file that was rewritten. The new aof file does not have these commands, so if there is an outage before the next rewrite occurs, you lose a lot of commands when you use AOF to recover the database.
    • solution : Set a aof rewrite buffer , which is only used during background process rewriting to write the database read and write commands that occur to the rewrite buffer (of course, The server process at this point writes to the rewrite buffer in addition to the database read and write commands that occur, writes to the AOF buffer to ensure normal aof operations, and then sends a signal to the server master process after the overriding child process finishes rewriting. The server master process appends the command in the AOF rewrite buffer to the new aof file, replacing the old aof file with the new AoF file.
  • Attention:
    • Why write these commands to the AOF buffer (because these commands write to the old aof file) during a background process rewrite when the database read-write commands that occur are written to the rewrite buffer?
      
      • If you can absolutely guarantee that the last master process replaces the old aof file with the new AoF file (which is not down until then), then writing to the AOF buffer is useless (because it will be overwritten), but if it goes down before, then our aof file is still an old aof file, This time the command is written to the AOF buffer to ensure that the AoF file saves as many commands as possible, in the future, to restore the database to a maximum of 1s of data lost. (Appendfsync everysec)
    • When the server master process appends commands from the AOF rewrite buffer to the new AoF file, the new client read-write command will be blocked during this operation (because the main process is doing the above). This is also the only part of the entire AOF rewrite process that is blocked .

Summarize:

• If both the RDB and the AOF are configured, the data persistence will take place, but when the data is recovered according to the file , the Rdb file is invalidated by the aof file.
  • Note: Data recovery is a blocking operation (any client read-write requests coming here are invalidated)
• Bgsave and bgrewriteaof (background aof Override) These two commands can not occur simultaneously
  • If Bgsave is executing, the arrival bgrewriteaof is executed after Bgsave executes
  • If bgrewriteaof in execution, here comes the Bgsave discard
• The RDB and AOF can be configured at the same time, but the last time the database is restored is the aof file.

Tenth Redis Persistence--rdb+aof