Enable Redis failover automatic failover via keepalived

Enable Redis failover automatic failover via keepalived [practice sharing]

Resources:

Http://patrick-tang.blogspot.com/2012/06/redis-keepalived-failover-system.html

http://deidara.blog.51cto.com/400447/302402

Currently, Redis does not have an official HA scenario similar to MySQL proxy or Oracle RAC.
The Redis author has a plan called Redis Sentinel (Http://redis.io/topics/sentinel), which is said to have three major functions: monitoring, alerting, and automatic failover, very good.
But unfortunately it is not possible to complete the development in the short term.

Therefore, how to automatically transfer in the event of a failure is a problem that needs to be solved.

Through the search for some information on the Internet, it is recommended to use Haproxy or keepalived to achieve, in fact, if it is to do failover rather than load balancing, keepalived efficiency is certainly more than Haproxy, So I decided to adopt the keepalived scheme.

Environment Introduction:
master:10.6.1.143
slave:10.6.1.144
Virtural IP Address (VIP): 10.6.1.200

Design ideas:
When Master and Slave are working normally, Master is responsible for service, Slave is responsible for standby;
When Master hangs up, Slave is normal, Slave takes over the service and shuts down the master-slave copy function;
When Master returns to normal, the data is synchronized from slave, the master-slave replication function is turned off after the data is synchronized, and the master identity is restored, while slave waits for the master synchronization data to complete before resuming slave identity.
Then loop in turn.

It is important to note that this requires a localization strategy on both master and slave, otherwise, in the automatic switching process, the other party's data will be emptied by the non-open side, resulting in a complete loss of data.

Below, is the specific implementation steps:

Installing keepalived on Master and slave
$ sudo apt-get install keepalived

Modify the/etc/hosts file for master and slave
$ sudo vim/etc/hosts

127.0.0.1localhost 10.6.1.143redis 10.6.1.144redis-slave

There is no configuration file after the default installation is complete keepalived, so we need to create it manually:

First, create the following configuration file on master:
$ sudo vim/etc/keepalived/keepalived.conf

Vrrp_script Chk_redis {Script "/etc/keepalived/scripts/redis_check.sh" # # #监控脚本 interval 2 # # #监控时间} vrrp_instance vi_1 { State MASTER # # #设置为MASTER Interface eth0 # # #监控网卡 virtual_router_id 101 # # # #权重值 Authentication {auth_type PASS # # #加密 Auth_pass redis # #密码} track_script {Chk_redis # # #执行上面定义的chk_redis} virtual_ipaddress {10.6.1.200 # # #VIP} notif y_master/etc/keepalived/scripts/redis_master.sh notify_backup/etc/keepalived/scripts/redis_backup.sh Notify_ Fault/etc/keepalived/scripts/redis_fault.sh notify_stop/etc/keepalived/scripts/redis_stop.sh}

Then, create the following configuration file on slave:
$ sudo vim/etc/keepalived/keepalived.conf

Vrrp_script Chk_redis {Script "/etc/keepalived/scripts/redis_check.sh" # # #监控脚本 interval 2 # # #监控时间} vrrp_instance vi_1 { State BACKUP # # # #设置为BACKUP Interface eth0 # # #监控网卡 virtual_router_id # # # # # #比MASTRE权重值低 Authentication {auth_ty PE PASS auth_pass redis # #密码与MASTRE相同} track_script {Chk_redis # #执行上面定义的chk_redis} virtual_ipaddress {10.6.1.200 # # # VIP} notify_master/etc/keepalived/scripts/redis_master.sh notify_backup/etc/keepalived/scripts/redis_backup.sh Notify_fault/etc/keepalived/scripts/redis_fault.sh notify_stop/etc/keepalived/scripts/redis_stop.sh}

Creating a script to monitor Redis on master and slave
$ sudo mkdir/etc/keepalived/scripts
$ sudo vim/etc/keepalived/scripts/redis_check.sh

#!/bin/bash alive= '/opt/redis/bin/redis-cli PING ' if ["$ALIVE" = = "PONG"]; Then echo $ALIVE exit 0 Else Echo $ALIVE exit 1 fi

Write the following key scripts that are responsible for the operation:
notify_master/etc/keepalived/scripts/redis_master.sh
notify_backup/etc/keepalived/scripts/redis_backup.sh
notify_fault/etc/keepalived/scripts/redis_fault.sh
notify_stop/etc/keepalived/scripts/redis_stop.sh

Because the keepalived is called according to the state when the state is converted:
Notify_master is called when the master State is entered
Notify_backup is called when the backup status is entered
Enter fault status call Notify_fault when abnormal conditions are found
Call Notify_stop when the keepalived program terminates

First, create the Notity_master and Notify_backup scripts on the Redis master:
$ sudo vim/etc/keepalived/scripts/redis_master.sh

#!/bin/bash rediscli= "/opt/redis/bin/redis-cli" logfile= "/var/log/keepalived-redis-state.log" echo "[Master]" > > $LOGFILE date >> $LOGFILE echo "Being master ..." >> $LOGFILE 2>&1 echo "Run slaveof cmd ..." > > $LOGFILE $REDISCLI slaveof 10.6.1.144 6379 >> $LOGFILE 2>&1 sleep #延迟10秒以后待数据同步完成后再取消同步状态 echo "Run S Laveof no one cmd ... ">> $LOGFILE $REDISCLI slaveof no one >> $LOGFILE 2>&1

$ sudo vim/etc/keepalived/scripts/redis_backup.sh

#!/bin/bash rediscli= "/opt/redis/bin/redis-cli" logfile= "/var/log/keepalived-redis-state.log" echo "[Backup]" > > $LOGFILE date >> $LOGFILE echo "Being slave ..." >> $LOGFILE 2>&1 sleep #延迟15秒待数据被对方同步完成之后再切换主从 Role echo "Run slaveof cmd ..." >> $LOGFILE $REDISCLI slaveof 10.6.1.144 6379 >> $LOGFILE 2>&1

Next, create the Notity_master and Notify_backup scripts on the Redis slave:

$ sudo vim/etc/keepalived/scripts/redis_master.sh

#!/bin/bash rediscli= "/opt/redis/bin/redis-cli" logfile= "/var/log/keepalived-redis-state.log" echo "[Master]" > > $LOGFILE date >> $LOGFILE echo "Being master ..." >> $LOGFILE 2>&1 echo "Run slaveof cmd ..." > > $LOGFILE $REDISCLI slaveof 10.6.1.143 6379 >> $LOGFILE 2>&1 sleep #延迟10秒以后待数据同步完成后再取消同步状态 echo "Run S Laveof no one cmd ... ">> $LOGFILE $REDISCLI slaveof no one >> $LOGFILE 2>&1

$ sudo vim/etc/keepalived/scripts/redis_backup.sh

#!/bin/bash rediscli= "/opt/redis/bin/redis-cli" logfile= "/var/log/keepalived-redis-state.log" echo "[Backup]" > > $LOGFILE date >> $LOGFILE echo "Being slave ..." >> $LOGFILE 2>&1 sleep #延迟15秒待数据被对方同步完成之后再切换主从 Role echo "Run slaveof cmd ..." >> $LOGFILE $REDISCLI slaveof 10.6.1.143 6379 >> $LOGFILE 2>&1

Then create the same script as the following in master and slave:
$ sudo vim/etc/keepalived/scripts/redis_fault.sh

#!/bin/bash logfile=/var/log/keepalived-redis-state.log echo "[Fault]" >> $LOGFILE date >> $LOGFILE

$ sudo vim/etc/keepalived/scripts/redis_stop.sh

#!/bin/bash logfile=/var/log/keepalived-redis-state.log echo "[Stop]" >> $LOGFILE date >> $LOGFILE

Add executable permissions to the script:
$ sudo chmod +x/etc/keepalived/scripts/*.sh

After the script has been created, we begin to test it in the following process:
1. Start Redis on Master
$ Sudo/etc/init.d/redis Start

2. Start Redis on the slave
$ Sudo/etc/init.d/redis Start

3. Start the keepalived on master
$ sudo/etc/init.d/keepalived Start

4. Start the keepalived on the slave
$ sudo/etc/init.d/keepalived Start

5. Try to connect Redis via VIP:
$ redis-cli-h 10.6.1.200 INFO

The connection was successful and slave was connected.
Role:master
Slave0:10.6.1.144,6379,online

6. Try inserting some data:
$ redis-cli-h 10.6.1.200 SET Hello Redis
Ok

Read data from VIP
$ redis-cli-h 10.6.1.200 GET Hello
"Redis"

reading data from Master
$ redis-cli-h 10.6.1.143 GET Hello
"Redis"

reading data from slave
$ redis-cli-h 10.6.1.144 GET Hello
"Redis"

Below, a simulated failure occurs:
Kill the Redis process on master:
$ sudo killall-9 redis-server

View the keepalived log on Master
$ tailf/var/log/keepalived-redis-state.log
[Fault]
Thu Sep 08:29:01 CST 2012

At the same time the log on slave shows:
$ tailf/var/log/keepalived-redis-state.log
[Master]
Fri Sep 14:14:09 CST 2012
Being master ....
Run slaveof cmd ...
Ok
Run slaveof NO One cmd ...
Ok

Then we can see that slave has taken over the service and assumed the role of master.
$ redis-cli-h 10.6.1.200 INFO
$ redis-cli-h 10.6.1.144 INFO
Role:master

Then we restore the master Redis process
$ Sudo/etc/init.d/redis Start

View the keepalived log on Master
$ tailf/var/log/keepalived-redis-state.log
[Master]
Thu Sep 08:31:33 CST 2012
Being master ....
Run slaveof cmd ...
Ok
Run slaveof NO One cmd ...
Ok

At the same time the log on slave shows:
$ tailf/var/log/keepalived-redis-state.log
[Backup]
Fri Sep 14:16:37 CST 2012
Being slave ....
Run slaveof cmd ...
Ok

You can see that the current master has resumed the master role again, and failover and automatic recovery have been successful.

Excerpt from: http://heylinux.com/archives/1942.html

Enable Redis failover automatic failover via keepalived