Redis master-Slave automatic failover

There is a problem with the Redis master-slave architecture persistence, that is, the last Test concludes that persistence needs to be configured on the master instance to ensure that data is not lost, so that the primary instance is bound to cause the I/O wait for the disk in the process of persisting the data to the hard disk, after the actual test, This persistent write drive process is unbearable for the application, so in most scenarios, the persistence configuration is considered to be on the slave instance, and when the primary instance is down, it will be promoted from instance to primary instance by manual or automatic, and continue to provide service! When the primary instance is restored, the data is synchronized from the original instance and then restored to the original master-slave state after the synchronization is complete! To achieve this requirement, you need to have keepalive, on the one hand keepalive provides VIP, you can avoid modifying the application connection, while the Redis instance configuration file Monitoring Section also needs to be modified for the whole network monitoring On the other hand, KeepAlive schedule the script to monitor the status of the master and slave instances, switch according to the specific situation! This article will focus on the implementation of Redis master-slave automatic failover! using keepalive

Environment Introduction
Operating system versions are: rhel5.4 64bit
Redis version: 2.6.4
Redis instance ports are: 6379
Redis instance passwords are: 123
vip:192.168.1.120
Primary instance is Server11 (192.168.1.112)
From instance to Server12 (192.168.1.113, turn on snapshot persistence)

One: Install keepalive software, server11 after the installation is complete directly SCP to Server12 can be

    [Email protected] ~]# wget http://keepalived.org/software/keepalived-1.1.19.tar.gz      [[email protected] ~]# Tar- ZXVF. /tarbag/keepalived-1.1.19.tar.gz      [[email protected] ~]# CD keepalived-1.1.19/      [[email protected] ~]#. Configure--prefix=/usr/local/keepalived && make && make install

Two: Configure the master node server11 configuration file

    [[email protected] ~]# cat/usr/local/keepalived/etc/keepalived/keepalived.conf      ! Configuration File for keepalived           global_defs {       router_id lvs_devel      }           vrrp_script Monitor_redis {       script "/usr/local/scripts/redis_monitor.sh"       interval 2        weight 2        }           vrrp_instance vi_1{ State        MASTER       interface eth0       virtual_router_id wuyi       mcast_src_ip 192.168.1.112       Priority       advert_int 1       authentication {       auth_type PASS       auth_pass password_123      }       Track_script {       Monitor_redis      }       virtual_ipaddress {       192.168.1.120       }       Notify_fault  /usr/local/scripts/redis_fault.sh         notify_stop   /usr/local/scripts/redis_stop.sh             

Three: Configure the Server12 configuration file from the node

  [[email protected] ~]# cat/usr/local/keepalived/etc/keepalived/ Keepalived.conf! Configuration File for keepalived global_defs {router_id lvs_devel} vrrp_script monitor_r Edis {script "/usr/local/scripts/redis_monitor.sh" Interval 2 weight 2} vrrp_instanc E vi_1{State BACKUP interface eth0 virtual_router_id mcast_src_ip 192.168.1.113 Priori Ty Advert_int 1 Authentication {auth_type PASS auth_pass password_123} TRACK_SCRI PT {Monitor_redis} virtual_ipaddress {192.168.1.120} notify_master/usr/local/scrip ts/redis_master.sh notify_backup/usr/local/scripts/redis_backup.sh Notify_fault/usr/local/scripts/redis _fault.sh notify_stop/usr/local/scripts/redis_stop.sh}  

Four: Prepare the relevant scripts, the master and slave instances need to exist these scripts, while note that the script needs to be executed by the permissions

[email protected] ~]# cat/usr/local/scripts/redis_monitor.sh #!/bin/bash alive=$ (/usr/local/redis2/ Bin/redis-cli-h 192.168.1.112-p 6379-a 123 PING) If ["$ALIVE" = = "PONG"];          Then echo $ALIVE exit 0 Else Echo $ALIVE killall-9 keepalived Service Network restart exit 1 fi [[email protected] ~]# Sh/usr/local/scripts/redis_monitor . sh PONG [[email protected] ~]# cat/usr/local/scripts/redis_master.sh #!/bin/bash rediscli= "/usr/local/redis2/bin/redis-cli-h 192.168.1.112-p 6379-a 123" logfile= "/usr/local/redis2/var/keepalived-redis-st Ate.log "echo" [Master] >> $LOGFILE date >> $LOGFILE echo "Being master ..." >> ; $LOGFILE 2>&1echo "Run slaveof cmd ..." >> $LOGFILE $REDISCLI slaveof 192.168.1.113 6379 >> $LOGFILE 2& gt;&1sleep echo "Run slaveof no one cmd ..." >> $LOGFILE $REDISCLI slaveof no one >> $L Ogfile 2>&1[email protected] ~]# cat/usr/local/scripts/redis_backup.sh #!/bin/bash rediscli= "/usr /local/redis2/bin/redis-cli-h 192.168.1.112-p 6379-a 123 "logfile="/USR/LOCAL/REDIS2/VAR/KEEPALIVED-REDIS-STATE.L OG "echo" [Backup] >> $LOGFILE date >> $LOGFILE echo "Being slave ..." >> $LOG FILE 2>&1sleep echo "Run slaveof cmd ..." >> $LOGFILE $REDISCLI slaveof 192.168.1.113 6379 >& Gt $LOGFILE 2>&1[email protected] ~]# cat/usr/local/scripts/redis_stop.sh #!/bin/bash logfile= "/usr/local/red Is2/var/keepalived-redis-state.log "echo" [Stop] >> $LOGFILE date >> $LOGFILE [[Email Protec Ted] ~]# cat/usr/local/scripts/redis_fault.sh #!/bin/bash logfile= "/usr/local/redis2/var/keepalived-redis-state. Log "echo" [Fault] >> $LOGFILE date >> $LOGFILE

Five: Master and Slave instances start the keepalive process, test the VIP is normal (here to modify the Redis configuration file listening address is 0.0.0.0)

[[email protected] ~]#/usr/local/keepalived/sbin/keepalived-d-f/usr/local/keepalived/etc/keepalived/ keepalived.conf [[email protected] ~]# tail-f/var/log/messages Dec 09:25:49 server11 keepalived_healthcheckers [7710]: Configuration is using:5499 Bytes Dec 09:25:49 server11 keepalived_healthcheckers[7710]: Using Linkwatch ker  Nel netlink reflector ... Dec 09:25:49 server11 keepalived_vrrp[7712]: VRRP sockpool: [IfIndex (2), Proto (FDS), FD (12,13)] Dec 09:25:49 Server KEEPALIVED_VRRP[7712]: Vrrp_script (Monitor_redis) succeeded Dec 09:25:50 Server11 keepalived_vrrp[7712]: VRRP_ Instance (vi_1{) Transition to MASTER State Dec 09:25:51 server11 keepalived_vrrp[7712]: vrrp_instance (vi_1{) Entering  MASTER State Dec 09:25:51 server11 keepalived_vrrp[7712]: vrrp_instance (vi_1{) setting protocol VIPs. Dec 09:25:51 server11 keepalived_vrrp[7712]: vrrp_instance (vi_1{) sending gratuitous ARPs on eth0 for 192.168.1.120 De C 09:25:51 Server11 avahi-dAEMON[4519]: Registering new address record for 192.168.1.120 on eth0. Dec 09:25:51 server11 keepalived_healthcheckers[7710]: NetLink Reflector reports IP 192.168.1.120 added Dec 12 09:25:5 1 Server11 keepalived_vrrp[7712]: NetLink Reflector reports IP 192.168.1.120 added Dec 09:25:56 server11 KEEPALIVED_VR RP[7712]: vrrp_instance (vi_1{) sending gratuitous ARPs on eth0 for 192.168.1.120 [[email protected] ~]# IP a |grep 192 inet 192.168.1.112/24 BRD 192.168.1.255 scope global eth0 inet 192.168.1.120/32 scope Global eth0 [[email& Nbsp;protected] ~]#/usr/local/keepalived/sbin/keepalived-d-f/usr/local/keepalived/etc/keepalived/ keepalived.conf [[email protected] ~]# tail-f/var/log/messages Dec 09:26:55 server12 keepalived_healthcheckers  [3106]: Configuration is using:5595 Bytes Dec 09:26:55 server12 keepalived_vrrp[3108]: vrrp_instance (vi_1{) Entering BACKUP State Dec 09:26:55 server12 keepalived_healthcheckers[3106]: Using linkwatch kernel NetLink Reflector ... Dec 09:26:55 server12 keepalived_vrrp[3108]: VRRP sockpool: [IfIndex (2), Proto (FDS), FD (12,13)] Dec 09:26:55 Server KEEPALIVED_VRRP[3108]: Vrrp_script (Monitor_redis) succeeded [[email protected] ~]#/usr/local/redis2/bin/ Redis-cli-h 192.168.1.120-a 123 Info |grep-a 3 ' Replication ' # Replication role:master connected_slaves:1 slave0:19 2.168.1.113,6379,online

Six: The master instance writes the test data, the script writes 25 test data in principle, but because the Redis default concurrency number is not optimized, it will cause some write requests to fail, and eventually write 231,839 test data, the total memory size is about 25M. In the writing process can observe the master-slave instance of the persistent file changes, the main instance of the persistent file maintained at 30k, from the instance of the continuous expansion!

[email protected] ~]# cat test.sh #!/bin/bash rediscli= "/usr/local/redis2/bin/redis-cli-h 192.168.1 .120-a 123-n 1 SET "id=1 while ($ID<50001)) do instance_name= "i-2-$ID-vm"UUID= ' Cat/proc/sys/kernel/random/uuid ' private_ip_address=10. ' Echo"$RANDOM% 255 + 1" | bc '. ' echo ' $RANDOM% 255 + 1 "| bc '. ' echo ' $RANDOM% 255 + 1" | BC ' \ created
    = ' Date"+%y-%m-%d%h:%m:%s" ' $REDISCLI vm_instance: $ID: instance_name "$INSTANCE _name" $REDISCLI vm_instance: $ID: UUID "$UUID" $REDISCLI vm_instance: $ID:p rivate_ip_address "$PRIVATE _ip_address" $REDISCLI vm_instance: $ID: Crea Ted "$CREATED" $REDISCLI vm_instance: $INSTANCE _name:id "$ID" ID=$ (($ID + 1))Done [[email protected] ~]# sh test.sh [[email protected] redis2]#/usr/local/redis2/bin/redis-cli-h 192.168.1.120-a 123 Info |egrep ' Used_memory_peak_human|db1:keys ' used_memory_peak_human:24.98m db1 : Keys=231839,expires=0

Seven: Simulating the primary instance failure, observing the log output, verifying that the VIP is successfully taken over from the instance, and turning the instance into read-write mode

    [[email protected] ~]# killall-9 redis-server [[email protected] ~]# IP a |grep 192 inet 192 .168.1.112/24 BRD 192.168.1.255 Scope Global eth0 [[email protected] ~]# ps-ef |grep redis Root 1 5886 6458 0 09:49 pts/0 00:00:00 grep redis [[email protected] ~]# ps-ef |grep keep Root 16029 6 458 0 09:49 pts/0 00:00:00 grep keep [[email protected] ~]# tail-f/usr/local/redis2/var/keepaliv      Ed-redis-state.log [master] Wed Dec 09:48:52 CST Being master ....      Run slaveof cmd ...      OK already connected to specified master Run slaveof NO one cmd ... OK [[email protected] ~]# tail-f/var/log/messages Dec 09:48:51 server12 keepalived_vrrp[3108]: VR Rp_instance (vi_1{) Transition to MASTER State Dec 09:48:52 server12 keepalived_vrrp[3108]: vrrp_instance (vi_1{) En Tering MASTER State Dec 09:48:52 server12 keepalived_vrrp[3108]: vrrp_instance (vi_1{) setting protocol VIPs.      Dec 09:48:52 server12 keepalived_vrrp[3108]: vrrp_instance (vi_1{) sending gratuitous ARPs on eth0 for 192.168.1.120 Dec 09:48:52 server12 keepalived_vrrp[3108]: NetLink Reflector reports IP 192.168.1.120 added Dec 09:48:52 SE      Rver12 avahi-daemon[2921]: Registering new address record for 192.168.1.120 on eth0. Dec 09:48:52 server12 keepalived_healthcheckers[3106]: NetLink Reflector reports IP 192.168.1.120 added Dec 12 09: 48:57 server12 keepalived_vrrp[3108]: vrrp_instance (vi_1{) sending gratuitous ARPs on eth0 for 192.168.1.120 [[E mail protected] ~]# IP a |grep 192 inet 192.168.1.113/24 BRD 192.168.1.255 Scope global eth0 inet 1 92.168.1.120/32 scope Global eth0 [[email protected] ~]#/usr/local/redis2/bin/redis-cli-h 192.168.1.120-  A 123 Info |grep-a 3 ' Replication ' # Replication Role:master connected_slaves:0

    [[Email protected] ~]# sh test.sh      [[email protected] ~]#/usr/local/redis2/bin/redis-cli-h 192.168.1.120-a 123 Info |egrep            ' Used_memory_peak_human|db1:keys '      used_memory_peak_human:26.78m      

Nine: Recovery process of Master instance role, automatic recovery using shell script

 [[email protected] ~]# Ssh-keygen [[email protected] ~]# CD. ssh/[ [email protected]. ssh]# ssh-copy-id-i id_rsa.pub [email protected] [[email protected] ~]# cat/usr/l ocal/scripts/recover_mastart.sh #!/bin/sh alive=$ (/usr/local/redis2/bin/redis-cli-h 192.168.1.113-p 6379-a 1 PING) Mdb=/usr/local/redis2/master_dump.rdb Sdb=/usr/local/redis2/slave_dump.rdb if ["$ALIVE" = = " PONG "];           then echo $ALIVE SCP [email protected]: $SDB $MDB Else Echo $ALIVE Exit 1 fi/usr/local/redis2/bin/redis-server/usr/local/redis2/etc/redis.conf/usr/local/keepalived /sbin/keepalived-d-f/usr/local/keepalived/etc/keepalived/keepalived.conf [[email protected] ~]# chmod +x/usr/local/scripts/recover_mastart.sh [[email protected] ~]# Sh/usr/local/scripts/recover_mastar t.sh 

10: Verifying data integrity and master-slave role recovery

    [[email protected] ~]#/usr/local/redis2/bin/redis-cli-h 192.168.1.120-a 123 Info |egrep ' Used_memory_peak_huma N|db1:keys ' used_memory_peak_human:26.78m db1:keys=249925,expires=0 [[email protected] ~]#/usr/l Ocal/redis2/bin/redis-cli-h 192.168.1.120-a 123 Info |grep-a 3 ' Replication ' # Replication role:master C Onnected_slaves:1 slave0:192.168.1.113,6379,online [[email protected] ~]#/usr/local/redis2/bin/redis -cli-h 192.168.1.113-a 123 Info |grep-a 3 ' Replication ' # Replication Role:slave master_host:192.168.1.1 master_port:6379 [[email protected] ~]#/usr/local/redis2/bin/redis-cli-h 192.168.1.120-a 123 inf o |egrep ' used_memory_peak_human|db1:keys ' used_memory_peak_human:26.78m db1:keys=249925,expires=0 Master instance k eepalive log: [[email protected] ~]# tail-f/var/log/messages Dec 10:08:13 server11 keepalived_vrrp[20231 ]: VRRP sockpool: [IFindEX (2), Proto (FDS), FD (11,12)] Dec 10:08:13 server11 keepalived_vrrp[20231]: Vrrp_script (Monitor_redis) succeeded Dec 10:08:13 server11 keepalived_vrrp[20231]: vrrp_instance (vi_1{) Transition to MASTER state Dec 12 10:08:13 Server11 keepalived_vrrp[20231]: vrrp_instance (vi_1{) Received higher prio advert Dec 10:08:13 server11 keepalived _VRRP[20231]: vrrp_instance (vi_1{) Entering BACKUP State Dec 10:08:15 server11 keepalived_vrrp[20231]: Vrrp_instan CE (vi_1{) forcing a new MASTER election Dec 10:08:16 server11 keepalived_vrrp[20231]: vrrp_instance (vi_1{) Transit      Ion to MASTER State Dec 10:08:17 server11 keepalived_vrrp[20231]: vrrp_instance (vi_1{) Entering MASTER State      Dec 10:08:17 server11 keepalived_vrrp[20231]: vrrp_instance (vi_1{) setting protocol VIPs. Dec 10:08:17 server11 keepalived_healthcheckers[20230]: NetLink Reflector reports IP 192.168.1.120 added Dec 12 10 : 08:17 server11 keepalived_vrrp[20231]: vrrp_insTance (vi_1{) sending gratuitous ARPs on eth0 for 192.168.1.120 Dec 10:08:17 server11 keepalived_vrrp[20231]: Netli NK Reflector reports IP 192.168.1.120 added Dec 10:08:17 server11 avahi-daemon[4519]: Registering new address Reco           Rd for 192.168.1.120 on eth0. [[email protected] ~]# IP a |grep 192 inet 192.168.1.112/24 BRD 192.168.1.255 scope global eth0 ine      T 192.168.1.120/32 scope Global eth0 from instance keepalive log: [[email protected] ~]# tail-f/var/log/messages Dec 09:56:01 server12 Last message repeated 4 times Dec 10:08:13 server12 keepalived_vrrp[3108]: vrrp_inst Ance (vi_1{) Received Lower Prio advert, forcing new election Dec 10:08:13 server12 keepalived_vrrp[3108]: vrrp_ins Tance (vi_1{) sending gratuitous ARPs on eth0 for 192.168.1.120 Dec 10:08:15 server12 keepalived_vrrp[3108]: vrrp_i Nstance (vi_1{) Received higher prio advert Dec 10:08:15 server12 keepalived_vrrp[3108]: Vrrp_instaNCE (vi_1{) Entering BACKUP State Dec 10:08:15 server12 keepalived_vrrp[3108]: vrrp_instance (vi_1{) removing PROTOC      Ol VIPs. Dec 10:08:15 server12 keepalived_healthcheckers[3106]: NetLink Reflector reports IP 192.168.1.120 removed Dec 12 1 0:08:15 server12 keepalived_vrrp[3108]: NetLink Reflector reports IP 192.168.1.120 removed Dec 10:08:15 server12 a           VAHI-DAEMON[2921]: Withdrawing address record for 192.168.1.120 on eth0. Convert log from instance role: [[email protected] ~]# tail-f/usr/local/redis2/var/keepalived-redis-state.log [backup] W      Ed Dec 10:08:15 CST-Being slave ....      Run slaveof cmd ...  Ok

Redis master-Slave automatic failover