Redis Persistent Storage (AOF and RDB two modes)

There are 2 kinds of data storage modes in Redis: cache-only,persistence; Cache-only is only a "caching" service, data is not persisted, the data disappears after the service terminates, there will be no "data recovery" means in this mode, it is a kind of low security/high efficiency/easy to extend; Persistence is a persistent backup of the data in memory to a disk file. It can be restored after the service restarts, and the data is relatively secure in this mode.

For persistence persistent storage, Redis provides two methods of persistence: Redis DataBase (RDB) append-only file (for short aof)

In addition to these two methods, the Redis in the early version of the virtual memory method, has now been discarded. I. Overview of RDB

RDB The data is written to a temporary file at a point in time, and after the persistence is over, replace the last persisted file with this temporary file to achieve data recovery.
Advantages: using a separate subprocess for persistence, the main process does not perform any IO operations to ensure Redis performance
Disadvantage:Rdb is a period of time to be persisted, if the redis between the persistence of failure, data loss will occur. So this is more appropriate when the data requirements are not rigorous.

The point at which this execution data is written to a temporary file can be determined by the configuration itself, by configuring Redis to perform a RDB operation in n seconds if more than M key is modified . This operation is similar to saving all of the Redis data at this point in time, one snapshot data at the moment. All this persistence method is also commonly called snapshots.

RDB is opened by default, and the specific configuration parameters in redis.conf are as follows;

#dbfilename: Persisted data is stored locally in the file
dbfilename dump.rdb
#dir: The persisted data is stored in the local path, if it is in the/redis/redis-3.0.6/ REDIS-CLI under SRC, the data is stored in the current src directory
dir./
# #snapshot触发的时机, save <seconds> <changes>  
# The following is 900 seconds, at least one change operation will be snapshot  
# #对于此值的设置, need to be careful to assess the system's change operation density  
# #可以通过 "Save" "" To turn off snapshot function  
#save时间, The following means that the 1 keys are changed at interval 900s for persistent storage, 10 key300s are changed for storage, and 10,000 key60s are changed for storage.
Save 900 1 Save
10000
# #当snapshot时出现错误无法继续时, whether to block client "change operations", "error" may be due to disk full/disk failure/ OS level Exceptions  
stop-writes-on-bgsave-error Yes  
# #是否启用rdb文件压缩, the default is "Yes", compression often means "extra CPU consumption", It also means that the smaller file size and the shorter network transmission time  
rdbcompression Yes  

The timing of the snapshot trigger is a joint decision between "interval" and "Number of changes", which will trigger snapshot if 2 conditions are met, otherwise the "change times" will be added to the next "time interval". Client requests are not blocked during the snapshot process. Snapshot first writes the data to the temporary file, and when successful, the temporary file name is Dump.rdb.

To recover data using RDB:
Automatic persistent data is stored after DUMP.RDB. It is actually done by restarting the Redis service (the data is synchronized from the Dump.rdb when the Redis server is started)

The client uses the command to persist the Save store:

./redis-cli-h ip-p Port save
./redis-cli-h ip-p Port Bgsave

One is stored in the foreground, and one is stored in the background. My client is on the server, so there is no need to connect the other machines directly./REDIS-CLI Bgsave. Because Redis is using a main thread to handle all client requests, this approach blocks all client requests. So it is not recommended to use. Another point to note is that each snapshot persistence is a complete write of the memory data to the disk, not an incremental synchronization of dirty data only. If the amount of data is large, and more write operations, will inevitably cause a lot of disk IO operations, may seriously affect performance. ii. Overview of AoF

Append-only file, the "Action + data" is appended to the end of the action log file as a formatted instruction, and the actual data changes are made after the Append operation returns (has been written to a file or is about to be written), and the "log file" preserves all history of the operation When the server requires data recovery, you can replay the log file directly to restore all the procedures. AOF is relatively reliable, it and MySQL in Bin.log, Apache.log, zookeeper Txn-log almost the same. AoF file content is a string that is very easy to read and parse.
Advantages: You can maintain higher data integrity, if the time to set the append file is 1s, if the redis failure, the maximum loss of 1s data, and if the log write incomplete support redis-check-aof for log repair aof files are not rewrite before they are too large to be overridden by a file, you can delete some of the commands (such as the Flushall of the error).
Disadvantages:aof files are larger than RDB files and are slow to recover.

We can simply think that aof is just a log file, this file will only record "change operations" (such as Set/del, etc.), if the constant number of changes in the server will cause the AoF file is very large, meaning that after server failure, the process of data recovery will be very long; in fact, A data after many changes, will produce a number of aof records, in fact, as long as the preservation of the current state, the history of the operation record can be discarded, because the AOF persistence mode also associated with the "aof rewrite."
AOF's characteristics determine that it is relatively safe, and if you expect less data to be lost, you can use the AOF model. If the AoF file is being written and suddenly the server fails, it may cause the last record of the file to be incomplete, and you can either manually or programmatically detect and correct the incomplete records so that the aof file can be restored properly; If you have aof in your Redis persistence method, you need to detect the integrity of the AOF file before restarting the server after it fails.

AoF default shutdown, open method, modify configuration file Reds.conf:appendonly Yes

# #此选项为aof功能的开关, the default is "No", you can use "yes" to open the AOF function # #只有在 "yes", aof rewrite/file synchronization and other features will take effect appendonly Yes # #指  
Fixed aof file name Appendfilename appendonly.aof # #指定aof操作中文件同步策略, has three legal values: Always everysec No, default to Everysec Appendfsync everysec # #在aof-rewrite period, appendfsync whether to suspend file synchronization, "No" means "no respite", "yes" means "stay", the default is "no" No-appendfsync-on-rewrite No # # The minimum file size (MB,GB) that is triggered by the aof file rewrite, only if greater than this aof file is larger than this size will trigger rewrite, default "64MB", recommended "512MB" Auto-aof-rewrite-min-size 64MB # #相对于 "  
Last "rewrite, the percentage of aof file should be increased when this rewrite is triggered.  
# #每一次rewrite之后, Redis records the size of the new aof file at this time (for example, a), then when aof file grows to a * (1 + P) # #触发下一次rewrite, every time aof record is added, the size of the current AOF file will be detected. Auto-aof-rewrite-percentage 

AoF is a file operation, and for a server that is denser than a change operation, the load on the disk IO will be heavier; In addition, Linux takes a "deferred write" method for file operations, which means that not every write operation triggers the actual disk operation, but enters the buffer. When the buffer data reaches the threshold to trigger the actual write (there are other times), this is the Linux optimization of the file system, but this can be a potential problem, if the buffer is not flushed to disk, when the physical machine failure (such as power), This may result in the loss of the last or multiple AOF records. Through the above configuration file, you can learn that Redis provides 3 aof record sync options: always: Each aof record is immediately synchronized to the file, which is the safest way to think of more disk operations and blocking delays, and is IO expense. Everysec: Sync every second, performance and security are more moderate way, is Redis recommended way. If you experience a physical server failure, it is possible that the AOF record is lost (possibly partially lost) in the last second. No:redis does not call file synchronization directly, but to the operating system to deal with, the operating system can be based on the buffer fill/channel idle time, such as timing trigger synchronization; This is a common file operation mode. Performance is good, in the case of a physical server failure, the amount of data loss will be due to OS configuration.

In fact, we can choose too little, everysec is the best choice. If you are very concerned about the reliability of each data, it is recommended that you choose a "relational database".
The aof file grows, its size directly affects the time of recovery, and the history operations in AOF files can be discarded. aof Rewrite operation is the process of "compressing" aof files, of course, Redis does not use "based on the original aof file" to rewrite the way, but to take a similar way snapshot: Based on the copy-on-write, the full amount of traverse in-memory data, Then sequentially to the AoF file. So aof rewrite can correctly react to the state of the current memory data, which is exactly what we need; In the *rewrite process, the new change operations will still be written to the original aof file, and these new changes will be collected by Redis (buffer, In Copy-on-write mode, the most extreme may be that all keys are modified during this time, will consume twice times of memory, when the memory data are all written to the new AoF file, the collection of new changes will also be appended to the new AoF file, The new aof file will then be renamed as Appendonly.aof, and all subsequent operations will be written to the new aof file. If the failure occurs during the rewrite process, it will not affect the normal work of the original AoF file, and will switch the file only after the rewrite is complete, because the rewrite process is more reliable. *

The timing of triggering rewrite can be stated through configuration files, while Redis can be manually intervened through bgrewriteaof instructions.

Redis-cli-h ip-p Port Bgrewriteaof

Because the rewrite Operation/aof record synchronization/snapshot consumes disk Io,redis has adopted a "schedule" policy: Whether it is "manual intervention" or system triggering, snapshot and rewrite need to be executed one by one.

The AOF rewrite process does not block client requests. The system will open a subprocess to complete. three. Summary:

AoF and Rdb each have their own advantages and disadvantages, which are determined by their respective characteristics: 1 aof More secure, can be more timely synchronization of data to the file, but aof need more disk IO expenditure, aof file size, file content recovery several times relatively slow.
*2) snapshot, security is poor, it is "normal time" data backup and Master-slave data synchronization of the best means, file size is small, recovery several times faster.

You can specify one of them through a configuration file, or use them both (not recommended at the same time), or all disabled, and in a well-formed environment, master typically uses Aof,slave to use snapshot, primarily because master needs to first ensure data integrity, It is the first choice for data backup; slave provides read-only service (currently slave can only provide read services), its primary purpose is to respond quickly to client read requests, but if your Redis runs in poor network stability/physical environment, It is recommended that you master and slave both take aof, which reduces the time cost of "manual data Backup"/"Manual data Recovery" when switching between master and slave roles; If your environment is very good and the service needs to receive intensive write operations, Then recommend master to take snapshot, and slave to use AOF.