Redis Persistence and common failures

1190000004135982Redis master-slave replicationthe principle of Redis master-slave replication

When establishing a master-slave relationship, slave configuration slaveof <master_host> <master_port>. The slave server sends a sync command to the primary server. Master accepts and fork a process to execute the bgsave command. The command generates an RDB file and sends the full amount to the slave server, the slave server receives and loads the Rdb file, and the primary server sends the buffer's commands incrementally to the slave server, eventually keeping the data state from the server and the primary server consistent.

How the RDB Works

When Redis generates a Dump.rdb file, the work process is as follows

• Redis Master Process Fork a child process

• Fork out the sub-process dump the memory dataset into a temporary RDB

• When a child process finishes writing a temporary RDB file, Redis replaces the old Rdb file with a new Rdb file

How the AOF works

Aof:append only file. Whenever Redis executes a command that changes the dataset, the command is appended to the end of the aof file. When Redis restarts, the program can recover data through the AoF file

Persistent file monitoring

The most straightforward approach to redis monitoring, of course, is to use the Info command provided by the system to get the status report of the Redis system by simply executing the following command.

redis-cli info

RDB File Status Monitoring

Parameters associated with the Rdb file status monitoring

• Rdb_changes_since_last_save indicates the number of key changes since the last RDB save

• Rdb_bgsave_in_progress Indicates whether the Bgsave operation is currently in progress. is for 1

• Rdb_last_save_time timestamp of the last time the Rdb file was saved

• Rdb_last_bgsave_time_sec time elapsed for last save

• Rdb_last_bgsave_status the last saved state

• Rdb_current_bgsave_time_sec the time it took to save the Rdb file currently

AOF File Status Monitoring

Parameters related to aof file status monitoring

• aof_enabled aof file is enabled

• Aof_rewrite_in_progress indicates whether a write AoF file operation is currently in progress

• Aof_rewrite_scheduled

• Aof_last_rewrite_time_sec timestamp of last Write

• Aof_current_rewrite_time_sec:-1

• Aof_last_bgrewrite_status:ok last Write State

• Aof_last_write_status:ok last Write State

Viewing the Rdb file generation time

Before we optimize master, we can look at the persistence state of Redis in one of our production environments.

#PersistenceLoading: 0Rdb_changes_since_last_save: 116200Rdb_bgsave_in_progress: 1rdb_last_save_time:1448944451rdb_last_ Bgsave_status:okrdb_last_bgsave_time_sec :85rdb_current_bgsave_time_sec:33< Span class= "Hljs-tag" >aof_enabled:0aof_rewrite_in_progress< Span class= "Hljs-pseudo" >:0aof_rewrite_scheduled:0aof_last_rewrite_time_sec:-1aof_current_rewrite_ Time_sec:-1aof_last_bgrewrite_status :okaof_last_write_status:ok   

With the REDIS-CLI Info command, you can see the value of the "rdb_last_bgsave_time_sec" parameter,
This value indicates the time when the last Bgsave command was executed. In the case of disk IO quantification, the larger the amount of memory that Redis occupies,
This value is also greater. Usually "rdb_last_bgsave_time_sec" this time depends on two factors:

• The amount of memory that Redis occupies

• The write speed of the disk.

rdb_last_bgsave_time_sec:85This ID indicates the time when we last saved the dump RDB file. This time is limited by the two factors mentioned above.

When Redis is in the rdb_bgsave_in_progress state, view performance through the Vmstat command, getting the WA value high, which means the CPU is waiting for
IO request completed, one of our online applications Redis occupies about 5G of memory, that is, Redis generates about 5G dump.rdb files

Vmstat command

R B swpd Free buffCache si so bi boIn CS US sy ID WA St0402239122242680572200800200486483640544311633500302227962242680572205200164827224175019116335003022230022426805722092004024612304235681163350030220068224268057221240064403284304473721633400302189522242680572221600 100 48648 4966 5786 1 2 63 35 0 0 3 0  215356 2242680 5722256 0 0 0 66168 3546 4382 2 1 62 Span class= "Hljs-number" >35 0          

Through the above output, see Bgsave for IO performance impact is relatively large

Then how to solve the performance problems caused by the Rdb file, but also to ensure the purpose of data persistence

The usual design idea is to use the "replication" mechanism to solve: that is, Master does not open the RDB log and aof log, to ensure that master read and write performance. Slave, however, opens the RDB and aof for persistence, ensuring data persistence,

Establish master-slave replication steps and disaster recovery

On the test machine, I opened two instances with ports 6379 and 6380, respectively.

master: 172.16.76.232 6379slave:  172.16.76.232 6380

Modify Configuration

Modify Master's redis.conf

Close an RDB

# save 900 1# save 300 10# save 60 10000

Close AoF

appendonly no

Start Redis for master and slave, respectively

service redis start

Modify the slave configuration to point to the master server

redis-cli > slaveof 172.16.76.232 6379

To view the replication status of slave

redis-cli > info replication

Script to simulate populating data

#!/bin/bashid=1while (($ID <50001))Do REDIS-CLISet"My:$ID " "Aaa_okthisis_omb5eviwbgphgbrj64raygperlkanhyb9slf_ $ID" REDIS-CLI set  "Your: $ID"  "Your_okthisis_omb5eviwbgphgbrj64raygperlkanhyb9slf_ $ID" REDIS-CLI set  "Her: $ID"  "Her_okthisis_omb5eviwbgphgbrj64raygperlkanhyb9slf_ $ID" REDIS-CLI set  "His: $ID"  "His_okthisis_omb5eviwbgphgbrj64raygperlkanhyb9slf_ $ID" ID=$ ( Span class= "hljs-variable" > $ID +1)) done    

Kill the master instance to simulate a disaster

master redis > killall -9 redis-server

SLAVE redis > SLAVEOF NO ONE

Cancels the slave synchronization, avoids the main library restarts before the incomplete data restores, thus directly overwrites the data from the library, causes all data loss.

Copy the Rdb and aof on slave to the master Data folder

/data/redis_data_slave/dump.rdb /data/redis_data/cp /data/redis_data_slave/Append.AOF /data/redis_data/

Start an instance of master

master redis > dbsize

See if data is restored

Turn slave replication on again

slave redis > slaveof 172.16.76.232 6379

Failure case report Redis lost data case

Background introduction:

我们的一台redis服务器，硬件配置为4核，4G内存。redis持久话方案是RDB。前面几个月redis使用的

Memory is around 1G. After a reboot, Redis recovers only part of the data, and then looks at the Redis.log file. See the following error

[23635] 25 Jul 08:30:54.059 * 10000 changes in 60 seconds. Saving...[23635] 25 Jul 08:30:54.059 # Can‘t save in background: fork: Cannot allocate memory

At this point, I think of the Redis boot warning

set to 0!Background save may fail under low memory condition.To fix this issue add ‘vm.overcommit_memory = 1‘ to /etc/sysctl.conf andthen reboot or run the command ‘sysctl vm.overcommit_memory=1‘ for this to take effect.

Translation

警告：过量使用内存设置为0！在低内存环境下，后台保存可能失败。为了修正这个问题，请在/etc/sysctl.conf 添加一项 ‘vm.overcommit_memory = 1‘ ，然后重启（或者运行命令‘sysctl vm.overcommit_memory=1‘ ）使其生效。

Vm.overcommit_memory Different value Descriptions

• 0 means to check if there is enough memory available, if yes, to allow allocations, and if memory is insufficient, reject the request and return an error to the application.

• 1 allow allocation of requests that exceed physical memory plus swap memory

• 2 Kernel always returns true

The data writeback mechanism of Redis is divided into two kinds

• Synchronous write-back is the Save command. The Redis master process writes data directly to disk. When the amount of data is large, this command will block, the response time is long

• The asynchronous write-back is the Bgsave command. The Redis Master Process fork A child process, copying the memory of the main process and writing back data to disk through the child process.

Fork a child process because the Rdb file is written. Equivalent to copying a memory image. The system's memory is 4G, and Redis takes up
Nearly 3G of memory, so it is certainly reported that memory cannot be allocated. If "vm.overcommit_memory" is set to 0, in case of low available memory
, the new memory cannot be allocated. If "vm.overcommit_memory" is set to 1. Then Redis will use swap memory.

Workaround:

• Method One: Modify the kernel parameter vi/etc/sysctl. Set up vm.overcommit_memory = 1 and then execute

  ``` sysctl -p ```

• Method two: Using swap memory is not a perfect solution. The best way is to expand physical memory.

Issues that may be encountered with replication

Data synchronization is not visible for a long time after using the slaveof command. The replication function failed, or the configuration was wrong. In fact, don't worry, there are two ways to determine whether replication is being established.

When you create a redis copy, you may initially find that slave does not start synchronizing data for long periods of time, and that the amount of data may be too large to cause master to be dump data slowly, if you perform "top-p $ (pgrep-d, redis-server) command on Master , you can see the dump process.

Way one: through the "top" command

[Root@img1_u ~]# Top-p $ (pgrep-d, redis-server) Top-14:06:UpIn the days,6:13,1 user, load average:1.18,1.32,1.20Tasks:2 Total,1 Running,1 sleeping,0 stopped,0 Zombiecpu (s):15.2%us,1.7%sy,0.6%ni,81.9%id,0.2%wa,0.0%hi,0.4%si,0.0%stmem:24542176k Total,22771848k used,1770328k free,2245720k Buffersswap:524280k Total, 0k used,524280k free,4369452k cached PID USER PR NI VIRT RES SHR S%cpu %mem time+ command21619 Root 20 0 5654 m 5.4g 388 R 99.9 23.0  0: 23.70 redis-server 1663 root 20 0 5654m 5.4g Span class= "Hljs-number" >1068 S 15.3 23.0 5042:31 redis-server         

Redis-server is a single process, and now there are 2 processes viewed through the "top" command, as previously mentioned, Redis, when establishing replication, will

Executes the BGSAVE command on the primary server. Fork a sub-process, dump the Rdb file. Master dump is finished, and then the snapshot file is passed to slave.

Way two: through "rdb_bgsave_in_progress" logo

Enter Master's REDIS-CLI

  redis-cli > info persistence ... loading:0rdb_changes_since_last_save< Span class= "Hljs-pseudo" >:0rdb_bgsave_in_progress:1rdb_last_save_time:1448992510rdb_last_bgsave_ Status:okrdb_last_bgsave_time_sec:4 rdb_current_bgsave_time_sec:1 ...      

If "rdb_bgsave_in_progress" is 1, then Master is in the Bgsave command. Meanwhile "rdb_current_bgsave_time_sec"
Displays the time that the Bgsave command has been executed. Since the RDB and AOF logs are not turned on by default on the master server, if the "rdb_bgsave_in_progress" is 1, then it is certain that a "bgsave" instruction is sent out of the Rdb file due to the replication reason.

Redis memory reaches Upper limit

Have the operation colleague response, the system is logged in the case, the operation will jump to the login page for no reason. Because our system does a distributed
Session, the default is to put the session into Redis, in accordance with previous failure experience. The maximum memory limit may be used by Redis
Causes the key to fail to be set. Log on to the Redis server to view the redis.conf file set maximum 8G memory "maxmemory 8g"
The current memory usage is then queried through "REDIS-CLI info memory" "used_memory_human:7.71g"
The values are then set by the Redis-cli tool. Error "oom command not allowed when used memory". Again
Verify that the Redis server has reached the maximum memory

Workaround:

1. Turn off Redis serverredis-cli shutdown

2. Modify the maximum memory "maxmemory" of a configuration file

3. Start Redis Serverredis-server redis.conf

• Published on December 12, 2015

Redis Persistence and common failures