Redis Persistence mode

For persistence persistent storage, Redis provides two methods of persistence:

• Redis DataBase (RDB)
  • Execution mechanism: snapshot, storing the binary form of key-value in databases directly in the Rdb file
  • Advantage: High performance (because it is a snapshot, and the execution frequency is lower than aof, and the Rdb file is stored directly in the binary form of the key-values, also fast for recovering data)
  • Using a separate subprocess for persistence, the master process does not perform any IO operations, guaranteeing the high performance of Redis
  • Disadvantage: If there is a downtime between the save configuration conditions, the data will be lost
  • The RDB is persisted over a period of time, and data loss occurs if Redis fails between persistence. So this approach is better suited to the time when data requirements are not rigorous.
• Append-only file (abbreviated as AOF)
  • Execution mechanism: Appends each modification command to the data to the AoF file
  • Pros: Data is not easy to lose
  • Can maintain higher data integrity, if the time to set append file is 1s, if redis fails, up to 1s of data will be lost, and if log write is not complete support redis-check-aof for log repair aof files are not rewrite (when the file is too large, the command is merged and rewritten), you can delete some of the commands (such as the wrong operation of the Flushall)
  • Disadvantage: Low performance (each modification operation is appended to the AoF file, the execution frequency is higher than the RDB, and the aof file is stored in the command, for the recovery of data requires line-by-row execution command, so recovery slow)
  • The aof file is larger than the Rdb file and is slow to recover.

In addition to these two methods, Redis has a method of virtual memory in the early version and is now obsolete.

I. Overview of the RDB

An RDB is a time when data is written to a temporary file, and after persistence, the temporary file is used to replace the last persisted file to achieve data recovery.

The point at which this execution data is written to a temporary file can be determined by configuration, by configuring Redis to perform an RDB operation if more than M key is modified within n seconds . This operation is similar to saving all Redis data at this point in time, one snapshot of the data. All of this persistence method is also commonly called snapshots.

The RDB is turned on by default, and the specific configuration parameters in redis.conf are as follows;

1.   #dbfilename: persisted data is stored locally in the file
2.   Dbfilename dump.rdb
3.   #dir: persisted data is stored locally in the path, if it is in/redis/redis-3.0.6/ SRC starts with REDIS-CLI, the data is stored in the current src directory
4.  dir./
5.  ## Snapshot trigger Time, save <seconds> <changes>
6.  # #如下为900秒后, there is at least one change operation before snapshot
7.  # #对于此值的设置, it is prudent to evaluate the system's change operation intensity
8.  # #可以通过" Save "" to turn off snapshot functionality (that is, no RDB is used)

    Code class= "Hljs Vala python" > #save时间, The following respectively indicates that 1 keys were changed at 900s for persistent storage, 10 key300s were changed for storage, and 10,000 key60s were changed for storage.      save 900 Span class= "Hljs-number" >1    save 300 10      save 60 10000     

1.  # #当snapshot时出现错误无法继续时, whether to block the client "change operation", "error" may be because the disk is full /disk failure/os level exception, etc.
2.  # When the background RDB process exports a snapshot (part of the Key-value) to an RDB file, the process is faulted (that is, when the last background save failed),
3.  # Whether the Redis master process also accepts write data to the database
4.   #该种方式会让用户知道在数据持久化到硬盘时出错了 (equivalent to a monitoring);
5.   #如果安装了很好的redis持久化监控, can be set to "no"  

stop-writes-on-bgsave-error yes  ##是否启用rdb文件压缩，默认为“yes”，压缩往往意味着“额外的cpu消耗”，同时也意味这较小的文件尺寸以及较短的网络传输时间 (如果希望RDB进程节省一点CPU时间，设置为no，但是可能最后的rdb文件会很大)

1. Rdbcompression Yes
2. #在redis重启后 Whether the Rdb file is detected to be corrupt (based on checksum check_sum in the Rdb file) before writing data from the Rdb file to memory
3. Rdbchecksum Yes

Snapshot trigger Time, there is "interval" and "Number of changes" together to determine, while meeting 2 conditions to trigger the snapshot, otherwise "change" will continue to accumulate to the next "interval". The snapshot process does not block client requests. Snapshot first writes the data to a temporary file, and when it finishes successfully, the temporary file is named Dump.rdb.

• If any of the above three save commands occur, the Bgsave command will occur, and there are two problems, assuming that 10,000 keys in 60s have changed (write, delete, update), will it be persisted immediately? After this persistence, assume that there is another 240s, and there is no change in the key operation, at this time if there is a persistence (because the 10 key changes to meet 300s, this is changed 10,000 key)?
  • Does not immediately persist: Redis defaults every 100ms using the Servercron function to check if the condition of the Save configuration is met, and the bgsave is satisfied, so that if within 100ms, I have satisfied the bgsave condition, Well, when I do bgsave, I'm going to wait until Servercron executes.
  • No more persistence: Redis has two parameters dirty (record the number of changes to key after the last Bgsave, the top of the example in 240s is 0), and Lastsave (the last time the Bgsave command was executed successfully), The number of modifications for each save configuration in the configuration refers to dirty, and each time period is calculated as the starting point for Lastsave.
• Commenting out all the save commands, the RDB will not work
• Rdbcompression YES: configured so that every string is compressed once it is stored in an RDB file?
  • Not: After set to Yes, compression occurs only if the length of the string is greater than or equal to 21 bytes
• Rdbchecksum Yes: Where is this checksum stored? Why is it possible to determine if a file is damaged by a comparison checksum?
  • The checksum (check_sum) is stored in the last eight bytes of the Rdb file (the detailed RDB file structure, see "Redis This based on implementation", "chapter 10th RDB Persistence"), the simple RDB file structure is as follows:
    • The first five bytes of "REDIS" at the beginning of an RDB file are the criteria for judging whether a file is an Rdb file (similar to the "magic number" in a class file)
    • Next 4 bytes: Rdb file version number (db_version)
    • Databases (note is plural): This stores key-value information stored in each library Redisdb (the core of the entire data persistence and recovery)
    • EOF (1 bytes): End of Rdb file body
    • Check_sum (8 bytes): Test and, the value is calculated according to the front four points, the value is calculated at the time of persistence and written to the end of the Rdb file, when the data is recovered according to the Rdb file, a checksum is computed based on the four points in the Rdb file. Then match the contents of the check_sum (that is, the last eight bytes) in the current Rdb file, if the same, the description is not corrupted, if not the same, the first four parts have data corruption (that is, the file corruption)
• When the Redis server is started, Redis automatically detects if there is an RDB file (provided that there is no aof) and, if so, recovers the data based on the Rdb file, which blocks the client's read and write operations to Redis until the recovery data is complete

To recover data using an RDB:
Automatic persistent data is stored after DUMP.RDB. The actual need to restart the Redis service is complete (data is synchronized from Dump.rdb when the Redis server is started)

The client uses the command to persist the Save store:

1. ./redis-cli- H IP- p port save
2. ./redis-cli- H IP- p port bgsave

One is stored in the foreground and one is stored in the background. My client is on the server, so there is no need to connect other machines directly./REDIS-CLI Bgsave. Because Redis uses a main thread to process all client requests, this method blocks all client requests. So it is not recommended. It is also important to note that each snapshot persistence is a full write of the memory data to disk once, not the incremental synchronization of only dirty data. If the amount of data is large, and more write operations, it will inevitably cause a large number of disk IO operations, may seriously affect performance.

Ii. Overview of AoF

Append-only file, the "Operation + data" is appended to the end of the operation log file by the format instruction, after the Append operation returns (has been written to the file or is about to write), the actual data changes, "log file" saved all the history of the operation process When the server needs data recovery, you can replay this log file directly to restore all the operational procedures. AOF is relatively reliable, and it is similar to the Txn-log in Bin.log, Apache.log, and zookeeper in MySQL. The AoF file content is a string that is very easy to read and parse.

We can simply think that aof is a log file, this file will only record "change operations" (for example: Set/del, etc.), if the server continues to a large number of change operations, will cause the AoF file is very large, meaning that the server fails, the data recovery process will be very long; A data after several changes, will produce multiple AOF records, in fact, as long as the current state, the history of the operation record can be discarded, because the AOF persistence mode is associated with the "AOF rewrite".
The AOF feature determines that it is relatively safe, and if you expect less data loss, you can use AOF mode. If the AoF file is being written and suddenly the server fails, it is possible that the last record of the file is incomplete, and you can detect and correct the incomplete records by hand or by program, so that the recovery through the AoF file can be normal; If you have aof in your Redis persistence method, you will need to detect the integrity of the AOF file before restarting it after the server fails to fail.

AoF default off, open method, modify config file reds.conf:appendonly Yes

2. # #此选项为aof功能的开关, the default is "No", you can use "yes" to turn on the AOF function
4. # #只有在 "Yes", features such as aof rewrite/file sync will only take effect
6. AppendOnly Yes
10. # #aof file storage path and file name
12. Appendfilename appendonly.aof
16. # #指定aof操作中文件同步策略, there are three legal values: Always everysec No, default is Everysec
18. # #always Every command, sync it to the AoF file immediately (it's safe, but it's slow, because each command will do a disk operation) IO is a big expense.
20. # #everysec每秒将数据写一次到aof文件, Redis recommends the way. If a physical server failure is encountered, it is possible to cause the AOF record to be lost (possibly partially lost) in the last second.
22. # #no Write to the operating system, the operating system to determine the size of the buffer, unified write to the AoF file (fast, but low synchronization frequency, easy to lose data)
24. Appendfsync everysec
26. # #在aof-rewrite period, appendfsync whether to postpone file synchronization, "No" means "no delay", "yes" means "pause", the default is "no"
28. # When the RDB persists the data, the AOF operation stops at this time, and if yes it stops
30. # During the time of the stop, the executed commands are written to the memory queue, and when the RDB is persisted, the commands are written to the AoF file uniformly .
32. # This parameter is configured to take into account the low frequency of the RDB persistence execution, but it takes a long time to execute, while the AOF executes at a high frequency, with a short execution time,
34. # Two sub-processes (RDB child process, aof subprocess) are inefficient (two sub-processes are disk read and write)
36. # However, if you change to Yes, the result is that many commands have been written to the memory queue during this time because of the long execution time of the RDB persistence .
38. # Finally causes the queue to not fit so that the commands written to the aof file may be much less aof
40. # When recovering data, you will lose a lot of data based on the aof file recovery
42. # So, just choose No.
44. no-appendfsync-on-rewrite No
48. # #aof文件rewrite触发的最小文件尺寸 (MB,GB), only larger than this aof file is larger than this size is to trigger rewrite, default "64MB", recommended "512MB"
50. ## aof rewrite: Invert the in-memory data into a command and then rewrite these commands to the AoF file
52. # Overriding Purpose: Assume that we have 100 operations on the same key in memory, and finally the value of the key is
54. # Then there are 100 command logs in the AOF, so there are two drawbacks:
56. # 1) aof file is too large to occupy hard disk space 2) very slow to recover data based on AOF file (100 commands required)
58. # If we invert the key in memory into "set key 100" and then write to the aof file,
60. # Then the size of the AoF file will be greatly reduced and the data can be recovered quickly (only 1 commands required) based on the aof file
62. # Note: AoF overrides occur only if the following two constraints are to be met;
64. # Assuming there is no second, then in the early aof, as long as slightly add some data, it happens aof rewrite
66. # When the aof growth percentage is the original 100% (that is, twice times the original size, for example, the original is 100m, the next rewrite is when the aof file is 200m), aof rewrite
68. auto-aof-rewrite-min-size 64mb
72. # #相对于 "Last" rewrite, the percentage that the AoF file should grow when this rewrite is triggered.
74. # #每一次rewrite之后, Redis records the size of the "new aof" file at this point (for example, a), and then when the aof file grows to a * (1 + P)
76. # #触发下一次rewrite, each time the AOF record is added, the size of the current AOF file will be detected.
78. auto-aof-rewrite-percentage

AOF is a file operation, for a more intensive change operation of the server, it will inevitably cause the load of disk IO is increased, and Linux has a "deferred write" method for file operations, that is, not every write operation will trigger the actual disk operation, but into the buffer, When the buffer data reaches the threshold to trigger the actual write (there are other times), this is the Linux file system optimization, but this can be a hidden danger, if the buffer is not flushed to disk, when the physical machine fails (such as power loss), It is possible to cause the loss of the last one or more aof records. With the above configuration file, you can learn that Redis provides 3 aof record synchronization options:

• Always: Every aof record is immediately synced to the file, which is the safest way to think of more disk operations and blocking latency, which is a large IO expense.
• Everysec: Synchronous once per second, the performance and security of a more moderate way, is also the way Redis recommended. If a physical server failure is encountered, it is possible to cause the AOF record to be lost (possibly partially lost) in the last second.
• No:redis does not directly call the file synchronization, but to the operating system to handle, the operating system can be based on buffer fill situation/channel idle time timing trigger synchronization, this is a common way of file operation. Performance is good, in the case of physical server failure, the amount of data loss due to OS configuration.

In fact, we can choose too little, everysec is the best choice. If you are very concerned about the reliability of each data, it is recommended that you choose a "relational database".
The aof file grows, its size directly affects the time of "failback", and the historical operations in the AoF file can be discarded. AOF Rewrite operation is the process of "compressing" the AOF file, of course, Redis does not use "based on the original AOF file" to rewrite the way, but instead took a similar snapshot way: Based on copy-on-write, the full amount of traversing in-memory data, Then sequence them sequentially into the aof file. So aof rewrite can correctly reflect the current state of the memory data, which is what we need,*rewrite process, for the new change operation will still be written to the original aof file, and these new change operations will be collected by Redis (buffer, Copy-on-write mode, the most extreme may be that all keys are modified during this period, will consume twice times the memory, when the memory data is written to the new AoF file, the collection of new change operations will be appended to the new AoF file, The new aof file will then be renamed to Appendonly.aof, and all subsequent operations will be written to the new aof file. If a failure occurs during the rewrite process, the original aof file will not be affected, and the file will not be toggled until the rewrite is complete, because the rewrite process is more reliable. *

The timing of triggering the rewrite can be declared through the configuration file, while Redis can be manually intervened through the bgrewriteaof instruction.

redis-cli -h ip -p port bgrewriteaof

Because the rewrite Operation/aof record synchronization/snapshot consumes the disk Io,redis takes a "schedule" policy: either "human intervention" or a system trigger, snapshot and rewrite need to be executed one by one.

The AOF rewrite process does not block client requests. The system will open a sub-process to complete.

Three. Summary:

The advantages and disadvantages of aof and RDB are determined by their respective characteristics:

1) aof more secure, can be more timely synchronization of data to the file, but aof need more disk IO expenditure, aof file size is large, file content recovery several relatively slow.

* *) snapshot, the security is poor, it is "normal period" data backup and Master-slave data synchronization best means, file size is small, recovery several times faster

• If both the RDB and the AOF are configured, the data persistence will take place, but when the data is recovered according to the file, the Rdb file is invalidated by the aof file.
  • Note: Data recovery is a blocking operation (any client read-write requests coming here are invalidated)
• Bgsave and bgrewriteaof (background aof Override) These two commands can not occur simultaneously
  • If Bgsave is executing, the arrival bgrewriteaof is executed after Bgsave executes
  • If bgrewriteaof in execution, here comes the Bgsave discard
• The RDB and AOF can be configured at the same time, but the last time the database is restored is the aof file.

---------------------This article from xiaozhu0301 csdn blog, full-text address please click: 79021436?utm_source=copy

Redis Persistence mode