Redis data loss and resolution

data write-back mechanism for Redis

the data writeback mechanism of Redis is divided into two types, synchronous and asynchronous,

1. synchronous Writeback is the Save command, and the main process writes back data directly to the disk. In the case of large data can cause the system to feign death for a long time, so it is generally not recommended.
2. asynchronous writeback is the Bgsave command, after the main process fork, copy itself and write back to the disk through this new process, and the new process shuts itself down after the write-back is completed. Since this does not require the main process to block, the system will not feign animation, the general default will use this method.

Personal Feeling Method 2 the way to fork the main process is clumsy, but it seems to be the only way. Hot data in memory can be modified at any time, and the memory image to be saved on disk must be frozen. Freezing will result in suspended animation. Fork a new process is equivalent to replicating a memory image then, so that the main process does not need to freeze, as long as the child process on the operation.

a fork in a small memory process does not require much resources, but when the memory space of the process is in G, fork becomes a horrible operation. Not to mention the process of fork 14G memory on a 16G memory host? It will certainly be reported that memory cannot be allocated. More exasperating is, the more frequent changes in the host fork on the more frequent, the fork of the cost of the operation itself may not be much better than suspended animation.

after you find the reason, directly modify the/etc/sysctl.conf kernel parameter vm.overcommit_memory= 1

sysctl-p

The Linux kernel determines whether or not to release according to the parameter vm.overcommit_memory parameters.

1. If vm.overcommit_memory = 1, Direct release
2. vm.overcommit_memory = 0: Compares the amount of virtual memory allocated for this request and the current free physical memory of the system plus swap, deciding whether to release.
3. vm.overcommit_memory= 2: The process will compare all allocated virtual memory plus the virtual memory allocated by this request and the current free physical memory of the system plus swap to determine whether to release.

Redis Persistence Practice and disaster recovery simulation

Resources:
Redis Persistence http://redis.io/topics/persistence
Google Groups https://groups.google.com/forum/?fromgroups=#!forum/redis-db

first, the discussion and understanding of Redis persistence

There are two ways of Redis persistence: RDB and AOF

First, we should be clear about what is the use of persistent data, and the answer is for data recovery after a reboot.
Redis is an in-memory database, either an RDB or a aof, that is just a measure of its data recovery.
So Redis will read the RDB or the aof file and reload it into memory when it recovers with the RDB and aof.

An RDB is a snapshot snapshot store, which is the default persistence method.
Can be understood as semi-persistent mode, which is to save data to disk periodically according to certain policies.
The resulting data file is Dump.rdb, and the cycle of the snapshot is defined by the save parameter in the configuration file.
The following are the default snapshot settings:

Save 1    #当有一条Keys数据被改变时, 900 seconds refresh to disk once save   #当有10条Keys数据被改变时, 300 seconds to disk save 60 10000 # When 10,000 keys data is changed, 60 seconds are flushed to disk

The Redis Rdb file does not break because its write operation is performed in a new process.
When a new Rdb file is generated, the Redis-generated subprocess writes the data to a temporary file, and then renames the temporary file to an Rdb file by means of an atomic rename system call.
In this way, Redis's RDB files are always available whenever a failure occurs.

At the same time, Redis's Rdb file is also a part of the Redis master-slave synchronization implementation.
The first time slave is synchronized to master is:
Slave the sync request to master, master dumps the Rdb file, then transfers the Rdb file to slave, then master forwards the cached command to the slave, and the first synchronization is complete.
The second and subsequent synchronization implementations are:
Master sends a snapshot of the variable directly to each slave in real time.
However, no matter what causes slave and master to disconnect, the process of repeating the above two steps will be repeated.
The master-slave replication of Redis is based on the persistence of memory snapshots, as long as there is slave there will be memory snapshots.

Obviously, the RDB has its shortcomings, that is, once the database has a problem, then the data stored in our Rdb file is not entirely new.
All the data from the last Rdb file generation to Redis outage was lost.

AOF (append-only File) is more persistent than the RDB approach.
Because when using aof persistence, Redis appends each received write command to the file via the Write function, similar to MySQL's binlog.
When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands that are saved in the file.
The corresponding setting parameters are:
$ vim/opt/redis/etc/redis_6379.conf

AppendOnly Yes       #启用AOF持久化方式appendfilename appendonly.aof #AOF文件的名称, default to appendonly.aof# Appendfsync always # Each time you receive a write command to force the disk to write immediately, is the most guaranteed full persistence, but the speed is also the slowest, generally not recommended to use. Appendfsync everysec #每秒钟强制写入磁盘一次, a good compromise in terms of performance and durability, is the recommended way. # Appendfsync No     #完全依赖OS的写入, typically 30 seconds or so, the best performance but persistent most not guaranteed, not recommended.

The full persistence of aof also poses another problem, and the persistence file becomes bigger and larger.
For example, we call the INCR Test command 100 times, the file must save all 100 commands, but in fact, 99 is redundant.
Because you want to restore the state of the database, it is enough to save a set test 100 in the file.
To compress aof persistent files, Redis provides the bgrewriteaof command.
When this command is received, Redis will use a snapshot-like method to save the data in memory to a temporary file in the form of a command, and finally replace the original file to achieve control over the growth of the aof file.
Because it is the process of simulating a snapshot, the old aof file is not read while overriding the AoF file, but instead the entire in-memory database content is rewritten with a command to a new aof file.
The corresponding setting parameters are:
$ vim/opt/redis/etc/redis_6379.conf

No-appendfsync-on-rewrite Yes   #在日志重写时, do not command append operation, but only put it in the buffer, to avoid the addition of the command to the disk IO conflict. Auto-aof-rewrite-percentage #当前AOF文件大小是上次日志重写得到AOF文件大小的二倍时, automatically start a new log rewrite process. Auto-aof-rewrite-min-size 64MB  #当前AOF文件启动新的日志重写过程的最小值 to avoid frequent rewrites due to small file sizes when Reids is just started.

What do you choose? Here are some suggestions from the official:
In general, if you want to provide high data supportability, it is recommended that you use both persistence methods at the same time.
If you can accept a few minutes of data loss from a disaster, you can use only an RDB.
Many users use aof only, but we recommend that you use an RDB as well, since the RDB can take a full snapshot of the data at times and provide a faster restart.
Therefore, we hope to unify aof and RDB into a persistence model in the future (long-term plan).

In terms of data recovery:
The RDB has a shorter startup time for two reasons:
One is that each data in an RDB file has only one record, and does not have a record of multiple operations of the data, as in the case of aof logs. So every piece of data just needs to be written once.
Another reason is that the Rdb file storage format and the Redis data in memory encoding format is consistent, do not need to do data encoding work, so the CPU consumption is much smaller than the AOF log load.

second, disaster recovery simulation
Since persistent data is used for data recovery after a reboot, it is very necessary to perform such a disaster recovery simulation.
It is said that if the data is to be persisted and stability is to be ensured, half of the physical memory is recommended to be left blank. Because at the time of the snapshot, the sub-processes that fork out the dump operation will occupy the same memory as the parent process, the real copy-on-write, the performance impact and the memory consumption are relatively large.
At present, the usual design idea is to use the replication mechanism to compensate for aof, snapshot performance deficiencies, to achieve data persistence.
That is, master on snapshot and aof do not do, to ensure that master read and write performance, and slave on the same time to open snapshot and aof to persist, to ensure the security of data.

First, modify the following configuration on master:
$ sudo vim/opt/redis/etc/redis_6379.conf

#save 1 #禁用Snapshot #save 10000appendonly no #禁用AOF

Next, modify the following configuration on the slave:
$ sudo vim/opt/redis/etc/redis_6379.conf

Save 1 #启用Snapshotsave 10save 10000appendonly Yes #启用AOFappendfilename appendonly.aof #AOF文件的名称 # Appendfsync AL Waysappendfsync everysec #每秒钟强制写入磁盘一次 # appendfsync no  no-appendfsync-on-rewrite Yes   #在日志重写时, No command append operation auto-aof-rewrite-percentage #自动启动新的日志重写过程auto-aof-rewrite-min-size 64MB  #启动新的日志重写过程的最小值

Start master and slave, respectively
$/etc/init.d/redis Start

Confirm that the snapshot parameter is not started in master after boot is complete
Redis 127.0.0.1:6379> CONFIG GET Save
1) "Save"
2) ""

Then generate 250,000 data in master with the following script:
Dongguo@redis:/opt/redis/data/6379$ Cat redis-cli-generate.temp.sh

#!/bin/bashrediscli= "redis-cli-a slavepass-n 1 SET" Id=1while (($ID <50001)) do  instance_name= "i-2-$ID-VM"  uuid= ' Cat/proc/sys/kernel/random/uuid '  private_ip_address=10. ' echo ' $RANDOM% 255 + 1 "| BC '. ' echo ' $RANDOM% 255 + 1 "| BC '. ' echo ' $RANDOM% 255 + 1 "| BC '  created= ' date "+%y-%m-%d%h:%m:%s" '  $REDISCLI vm_instance: $ID: instance_name "$INSTANCE _name"  $ REDISCLI vm_instance: $ID: UUID "$UUID"  $REDISCLI vm_instance: $ID:p rivate_ip_address "$PRIVATE _ip_address"  $ REDISCLI vm_instance: $ID: Created "$CREATED"  $REDISCLI vm_instance: $INSTANCE _name:id "$ID"  id=$ (($ID + 1)) Done

Dongguo@redis:/opt/redis/data/6379$./redis-cli-generate.temp.sh

In the process of generating data, it is clear to see that master only creates the Dump.rdb file on the first slave synchronization, and then gives slave by the way of incremental transfer of commands.
Dump.rdb file no longer grows.
Dongguo@redis:/opt/redis/data/6379$ LS-LH
Total 4.0K
-rw-r--r--1 root root Sep 00:40 Dump.rdb

On the slave, you can see that dump.rdb files and aof files are growing, and the aof file growth rate is significantly larger than the Dump.rdb file.
[Email protected]:/opt/redis/data/6379$ LS-LH
Total 24M
-rw-r--r--1 root root 15M Sep 12:06 appendonly.aof
-rw-r--r--1 root root 9.2M Sep 12:06 Dump.rdb

After the data is inserted, the current amount of data is acknowledged first.
Redis 127.0.0.1:6379> Info

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version : 4.4.5PROCESS_ID:27623RUN_ID:E00757F7B2D6885FA9811540DF9DFED39430B642UPTIME_IN_SECONDS:1541UPTIME_IN_DAYS:0LRU _clock:650187used_cpu_sys:69.28used_cpu_user:7.67used_cpu_sys_children:0.00used_cpu_user_children:0. 00connected_clients:1connected_slaves:1client_longest_output_list:0client_biggest_input_buf:0blocked_clients:0 Used_memory:33055824used_memory_human:31.52mused_memory_rss:34717696used_memory_peak:33055800used_memory_peak_ Human:31.52mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:0changes_since_last_ Save:250000bgsave_in_progress:0last_save_time:1348677645bgrewriteaof_in_progress:0total_connections_received : 250007total_commands_processed:750019expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_ channels:0pubsub_patterns:0latest_fork_usec:246vm_enabled:0role:masterslave0:10.6.1.144,6379,onlinedb1:keys= 250000,expires=0

The current amount of data is 250,000 key, which consumes 31.52M of memory.

Then we kill the master Redis process directly, simulating the disaster.
Dongguo@redis:/opt/redis/data/6379$ sudo killall-9 redis-server

Let's check the status in slave:
Redis 127.0.0.1:6379> Info

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version : 4.4.5PROCESS_ID:13003RUN_ID:9B8B398FC63A26D160BF58DF90CF437ACCE1D364UPTIME_IN_SECONDS:1627UPTIME_IN_DAYS:0LRU _clock:654181used_cpu_sys:29.69used_cpu_user:1.21used_cpu_sys_children:1.70used_cpu_user_children : 1.23connected_clients:1connected_slaves:0client_longest_output_list:0client_biggest_input_buf:0blocked_ Clients:0used_memory:33047696used_memory_human:31.52mused_memory_rss:34775040used_memory_peak:33064400used_ Memory_peak_human:31.53mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:1changes_ Since_last_save:3308bgsave_in_progress:0last_save_time:1348718951bgrewriteaof_in_progress:0total_connections_ Received:4total_commands_processed:250308expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_ Channels:0pubsub_patterns:0latest_fork_usec:694vm_enabled:0role:slaveaof_current_size:17908619aof_base_size : 16787337aof_pending_rewrite:0aof_buffer_leNgth:0aof_pending_bio_fsync:0master_host:10.6.1.143master_port:6379master_link_status:downmaster_last_io_ seconds_ago:-1master_sync_in_progress:0master_link_down_since_seconds:25slave_priority:100db1:keys=250000, Expires=0

You can see that the status of Master_link_status is down and master is inaccessible.
At this point, the slave still works well and retains aof and RDB files.

Below we will restore the data on master through the saved aof and Rdb files on the slave.

First, the synchronization state on the slave is canceled to prevent the main library from restarting before the incomplete data is restored, thus directly overwriting the data from the library, resulting in all data loss.
Redis 127.0.0.1:6379> slaveof NO One
Ok

Confirm that there is no master-related configuration information:
Redis 127.0.0.1:6379> INFO

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version : 4.4.5PROCESS_ID:13003RUN_ID:9B8B398FC63A26D160BF58DF90CF437ACCE1D364UPTIME_IN_SECONDS:1961UPTIME_IN_DAYS:0LRU _clock:654215used_cpu_sys:29.98used_cpu_user:1.22used_cpu_sys_children:1.76used_cpu_user_children : 1.42connected_clients:1connected_slaves:0client_longest_output_list:0client_biggest_input_buf:0blocked_ Clients:0used_memory:33047696used_memory_human:31.52mused_memory_rss:34779136used_memory_peak:33064400used_ Memory_peak_human:31.53mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:1changes_ Since_last_save:0bgsave_in_progress:0last_save_time:1348719252bgrewriteaof_in_progress:0total_connections_ Received:4total_commands_processed:250311expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_ Channels:0pubsub_patterns:0latest_fork_usec:1119vm_enabled:0role:masteraof_current_size:17908619aof_base_size : 16787337aof_pending_rewrite:0aof_buffer_lenGth:0aof_pending_bio_fsync:0db1:keys=250000,expires=0 

To copy data files on slave:
[Email protected]:/opt/redis/data/6379$ Tar Cvf/home/dongguo/data.tar *
Appendonly.aof
Dump.rdb

Upload the Data.tar to master and try to recover the data:
You can see that the master directory has an initialized slave data file, which is small and is deleted.
[Email protected]:/opt/redis/data/6379$ ls-l
Total 4
-rw-r--r--1 root root Sep 00:40 Dump.rdb
[Email protected]:/opt/redis/data/6379$ sudo rm-f dump.rdb

Then unzip the data file:
[Email protected]:/opt/redis/data/6379$ sudo tar xf/home/dongguo/data.tar
[Email protected]:/opt/redis/data/6379$ LS-LH
Total 29M
-rw-r--r--1 root root 18M Sep 01:22 appendonly.aof
-rw-r--r--1 root root 12M Sep 01:22 Dump.rdb

Start Redis on master;
[Email protected]:/opt/redis/data/6379$ sudo/etc/init.d/redis Start
Starting Redis Server ...

To see if the data is restored:
Redis 127.0.0.1:6379> INFO

REDIS_VERSION:2.4.17REDIS_GIT_SHA1:00000000REDIS_GIT_DIRTY:0ARCH_BITS:64MULTIPLEXING_API: Epollgcc_version:4.4.5process_id:16959run_id:6e5ba6c053583414e75353b283597ea404494926uptime_in_seconds : 22uptime_in_days:0lru_clock:650292used_cpu_sys:0.18used_cpu_user:0.20used_cpu_sys_children:0.00used_cpu_user_ children:0.00connected_clients:1connected_slaves:0client_longest_output_list:0client_biggest_input_buf:0 Blocked_clients:0used_memory:33047216used_memory_human:31.52mused_memory_rss:34623488used_memory_peak : 33047192used_memory_peak_human:31.52mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_ Enabled:0changes_since_last_save:0bgsave_in_progress:0last_save_time:1348680180bgrewriteaof_in_progress:0total _connections_received:1total_commands_processed:1expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0 Pubsub_channels:0pubsub_patterns:0latest_fork_usec:0vm_enabled:0role:masterdb1:keys=250000,expires=0 

You can see that 250,000 of the data has been fully restored to master.

At this point, you can safely restore the synchronization settings of the slave.
Redis 127.0.0.1:6379> slaveof 10.6.1.143 6379
Ok

To view the synchronization status:
Redis 127.0.0.1:6379> INFO

Redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epollgcc_version : 4.4.5PROCESS_ID:13003RUN_ID:9B8B398FC63A26D160BF58DF90CF437ACCE1D364UPTIME_IN_SECONDS:2652UPTIME_IN_DAYS:0LRU _clock:654284used_cpu_sys:30.01used_cpu_user:2.12used_cpu_sys_children:1.76used_cpu_user_children : 1.42connected_clients:2connected_slaves:0client_longest_output_list:0client_biggest_input_buf:0blocked_ Clients:0used_memory:33056288used_memory_human:31.52mused_memory_rss:34766848used_memory_peak:33064400used_ Memory_peak_human:31.53mmem_fragmentation_ratio:1.05mem_allocator:jemalloc-3.0.0loading:0aof_enabled:1changes_ Since_last_save:0bgsave_in_progress:0last_save_time:1348719252bgrewriteaof_in_progress:1total_connections_ Received:6total_commands_processed:250313expired_keys:0evicted_keys:0keyspace_hits:0keyspace_misses:0pubsub_ Channels:0pubsub_patterns:0latest_fork_usec:12217vm_enabled:0role:slaveaof_current_size:17908619aof_base_size : 16787337aof_pending_rewrite:0aof_buffer_lenGth:0aof_pending_bio_fsync:0master_host:10.6.1.143master_port:6379master_link_status:upmaster_last_io_seconds_ Ago:0master_sync_in_progress:0slave_priority:100db1:keys=250000,expires=0

The master_link_status is displayed as up and the sync status is normal.

In the process of this recovery, we also copied the aof and RDB files, then which one of the files to complete the recovery of data?
In fact, when the Redis server is hung up, the data is restored to memory at the following priority level when restarted:
1. If only configure AOF, load aof file to recover data when reboot;
2. If both RDB and AOF are configured at the same time, startup is only load aof file recovery data;
3. If only the RDB is configured, startup is the load dump file to recover the data.

That is, the priority of the AOF is higher than the RDB, which is well understood, because the AOF itself guarantees the integrity of the data above the RDB.

In this case, we secured the data by enabling the AOF and Rdb on the slave, and restored the master.

However, in our current online environment, because the data are set to expire time, the use of aof is not very practical, too frequent write operations will cause the AOF file to grow to an unusually large, much more than the actual amount of our data, which will also lead to data recovery in a significant amount of time.
Therefore, you can only open snapshot on slave to localize, and you can consider the frequency of Save or call a scheduled task to perform periodic bgsave of snapshot storage to ensure the integrity of the localized data as much as possible.
In such a framework, if only master hangs, slave complete, data recovery can reach 100%.
If master and slave are hung at the same time, data recovery can also be achieved in an acceptable degree.

Redis data loss and resolution