Redis Persistence Practice and disaster recovery simulation (top)

Resources:
Redis Persistence http://redis.io/topics/persistence
Google Groups https://groups.google.com/forum/?fromgroups=#!forum/redis-db

first, the discussion and understanding of Redis persistence

There are two ways of Redis persistence: RDB and AOF

First, we should be clear about what is the use of persistent data, and the answer is for data recovery after a reboot.
Redis is an in-memory database, either an RDB or a aof, that is just a measure of its data recovery.
So Redis will read the RDB or the aof file and reload it into memory when it recovers with the RDB and aof.

An RDB is a snapshot snapshot store, which is the default persistence method.
Can be understood as semi-persistent mode, which is to save data to disk periodically according to certain policies.
The resulting data file is Dump.rdb, and the cycle of the snapshot is defined by the save parameter in the configuration file.
The following are the default snapshot settings:

Save 1    #当有一条Keys数据被改变时, 900 seconds refresh to disk once
save   #当有10条Keys数据被改变时, 300 seconds to disk
save 60 10000 # When 10,000 keys data is changed, 60 seconds are flushed to disk

The Redis Rdb file does not break because its write operation is performed in a new process.
When a new Rdb file is generated, the Redis-generated subprocess writes the data to a temporary file, and then renames the temporary file to an Rdb file by means of an atomic rename system call.
In this way, Redis's RDB files are always available whenever a failure occurs.

At the same time, Redis's Rdb file is also a part of the Redis master-slave synchronization implementation.
The first time slave is synchronized to master is:
Slave the sync request to master, master dumps the Rdb file, then transfers the Rdb file to slave, then master forwards the cached command to the slave, and the first synchronization is complete.
The second and subsequent synchronization implementations are:
Master sends a snapshot of the variable directly to each slave in real time.
However, no matter what causes slave and master to disconnect, the process of repeating the above two steps will be repeated.
The master-slave replication of Redis is based on the persistence of memory snapshots, as long as there is slave there will be memory snapshots.

Obviously, the RDB has its shortcomings, that is, once the database has a problem, then the data stored in our Rdb file is not entirely new.
All the data from the last Rdb file generation to Redis outage was lost.

AOF (append-only File) is more persistent than the RDB approach.
Because when using aof persistence, Redis appends each received write command to the file via the Write function, similar to MySQL's binlog.
When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands that are saved in the file.
The corresponding setting parameters are:
$ vim/opt/redis/etc/redis.conf

AppendOnly Yes       #启用AOF持久化方式
appendfilename appendonly.aof #AOF文件的名称, default is appendonly.aof
# Appendfsync Always #每次收到写命令就立即强制写入磁盘, is the most guaranteed full persistence, but the speed is also the slowest, generally not recommended to use.
Appendfsync everysec #每秒钟强制写入磁盘一次, a good compromise in terms of performance and durability, is the recommended way.
# Appendfsync No     #完全依赖OS的写入, typically 30 seconds or so, the best performance but persistent most not guaranteed, not recommended.

The full persistence of aof also poses another problem, and the persistence file becomes bigger and larger.
For example, we call the INCR Test command 100 times, the file must save all 100 commands, but in fact, 99 is redundant.
Because you want to restore the state of the database, it is enough to save a set test 100 in the file.
To compress aof persistent files, Redis provides the bgrewriteaof command.
When this command is received, Redis will use a snapshot-like method to save the data in memory to a temporary file in the form of a command, and finally replace the original file to achieve control over the growth of the aof file.
Because it is the process of simulating a snapshot, the old aof file is not read while overriding the AoF file, but instead the entire in-memory database content is rewritten with a command to a new aof file.
The corresponding setting parameters are:
$ vim/opt/redis/etc/redis.conf

No-appendfsync-on-rewrite Yes   #在日志重写时, do not command append operation, but only put it in the buffer, to avoid the addition of the command to the disk IO conflict.
auto-aof-rewrite-percentage #当前AOF文件大小是上次日志重写得到AOF文件大小的二倍时, automatically start a new log rewrite process.
auto-aof-rewrite-min-size 64MB  #当前AOF文件启动新的日志重写过程的最小值 to avoid frequent rewrites due to small file sizes when Reids is just started.

What are you going to choose? Here are some suggestions from the official:
In general, if you want to provide high data supportability, it is recommended that you use both persistence methods at the same time.
If you can accept a few minutes of data loss from a disaster, you can use only an RDB.
Many users use aof only, but we recommend that you use an RDB as well, since the RDB can take a full snapshot of the data at times and provide a faster restart.
Therefore, we hope to unify aof and RDB into a persistence model in the future (long-term plan).

In terms of data recovery:
The RDB has a shorter startup time for two reasons:
One is that each data in an RDB file has only one record, and does not have a record of multiple operations of the data, as in the case of aof logs. So every piece of data just needs to be written once.
Another reason is that the Rdb file storage format and the Redis data in memory encoding format is consistent, do not need to do data encoding work, so the CPU consumption is much smaller than the AOF log load.