Keepalived + redis for high-availability automatic failover

Keepalived + redis for high-availability automatic failover
Keepalived + redis for high-availability automatic failover
Install redis and keepalived on server A (10.0.11.2) and server B (10.0.12.2)
A is the default master and B is the slave (add SLAVEOF 10.0.11.2 6379 TO THE redis configuration file ).
Redis on A and B All enable localization policies. Appendonly yes

Contents of the keepalived configuration file for server

-------begin------! Configuration File for keepalivedglobal_defs {notification_email {hq@xxx.com}notification_email_from hq@xxx.comsmtp_server 127.0.0.1smtp_connect_timeout 30router_id redis1.xxx.com}vrrp_script chk_redis {script "/opt/redis/sh/redis_check.sh"interval 2}vrrp_instance VI_1 {state MASTERinterface bond0virtual_router_id 51priority 101advert_int 2authentication {auth_type PASSauth_pass 1111}track_script {chk_redis}virtual_ipaddress {10.0.11.0}notify_master /opt/redis/sh/redis_master.shnotify_backup /opt/redis/sh/redis_backup.shnotify_fault /opt/redis/sh/redis_fault.shnotify_stop /opt/redis/sh/redis_stop.sh}-----end-----

Note: Some emails in global_defs can be written as needed. To implement email notification, you must enter the actual information.
Script "/opt/redis/sh/redis_check.sh" # monitor script path
Interval 2 # monitoring frequency
State MASTER # default status
Interface bond0 # Nic name
Virtual_router_id 51 # A, B server can be set the same
Priority 101 # set a higher priority than B
Advert_int 2 # It seems that the broadcast frequency is unknown.
Authentication {# A, B server can be set the same
Auth_type PASS
Auth_pass 1111
}
Track_script {# monitoring name, as set above
Chk_redis
}
Virtual_ipaddress {# virtual IP address, which is used by the client to access redis
10.0.11.0
}
# The following is the script path executed in each status:
Notify_master/opt/redis/sh/redis_master.sh # become the master
Notify_backup/opt/redis/sh/redis_backup.sh # become backup
Notify_fault/opt/redis/sh/redis_fault.sh # When the monitoring script exit 1
Notify_stop/opt/redis/sh/redis_stop.sh # When the keepalived service is stopped
/Opt/redis/sh/redis_check.sh

--begin--#!/bin/bashALIVE=`/usr/local/bin/redis-cli PING`LOGFILE="/opt/redis/logs/keepalived-redis-state.log"if [ "$ALIVE" == "PONG" ]; thenecho $ALIVE#echo "check master pong" >> $LOGFILEexit 0elseecho $ALIVEexit 1fi--end--/opt/redis/sh/redis_master.sh--begin--#!/bin/bashREDISCLI="/usr/local/bin/redis-cli"LOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[master]" >> $LOGFILEdate >> $LOGFILEecho "Being master...." >> $LOGFILE 2>&1echo "Run SLAVEOF cmd ..." >> $LOGFILE$REDISCLI SLAVEOF 10.0.12.2 6379 >> $LOGFILE 2>&1sleep 15echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE$REDISCLI SLAVEOF NO ONE$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1--end--

/Opt/redis/sh/redis_backup.sh

--begin--REDISCLI="/usr/local/bin/redis-cli"LOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[backup]" >> $LOGFILEdate >> $LOGFILEecho "Being slave...." >> $LOGFILE 2>&1sleep 15echo "Run SLAVEOF cmd..." >> $LOGFILE$REDISCLI SLAVEOF 10.0.12.2 6379 >> $LOGFILE 2>&1--end--

/Opt/redis/sh/redis_fault.sh

--begin--#!/bin/bashLOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[fault]" >> $LOGFILEdate >> $LOGFILEsh /opt/redis/sh/redis_backup.sh--end--

/Opt/redis/sh/redis_stop.sh

--begin--#!/bin/bashLOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[stop]" >> $LOGFILEdate >> $LOGFILE--end--

Server B Configuration
Content of the keepalived configuration file

-------begin------! Configuration File for keepalivedglobal_defs {notification_email {hq@xxx.com}notification_email_from hq@xxx.comsmtp_server 127.0.0.1smtp_connect_timeout 30router_id redis2.xxx.com}vrrp_script chk_redis {script "/opt/redis/sh/redis_check.sh"interval 2}vrrp_instance VI_1 {state BACKUPinterface bond0virtual_router_id 51priority 99authentication {auth_type PASSauth_pass 1111}track_script {chk_redis}virtual_ipaddress {10.0.11.0}notify_master /opt/redis/sh/redis_master.shnotify_backup /opt/redis/sh/redis_backup.shnotify_fault /opt/redis/sh/redis_fault.shnotify_stop /opt/redis/sh/redis_stop.sh}-----end-----

/Opt/redis/sh/redis_check.sh

--begin--#!/bin/bashALIVE=`/usr/local/bin/redis-cli -h 10.0.11.2 -p 6379 PING`LOGFILE="/opt/redis/logs/keepalived-redis-state.log"if [ "$ALIVE" != "PONG" ]; thenecho $ALIVEexit 0elseecho $ALIVEexit 1fi--end--

/Opt/redis/sh/redis_master.sh

--begin--#!/bin/bashREDISCLI="/usr/local/bin/redis-cli"LOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[master]" >> $LOGFILEdate >> $LOGFILEecho "Being master...." >> $LOGFILE 2>&1##echo "master Run SLAVEOF 10.0.11.2 cmd ..." >> $LOGFILE##REDISCLI SLAVEOF 10.0.11.2 6379 >> $LOGFILE 2>&1#sleep 10echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1--end--

/Opt/redis/sh/redis_backup.sh

--begin--#!/bin/bashREDISCLI="/usr/local/bin/redis-cli"LOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[backup]" >> $LOGFILEdate >> $LOGFILEecho "Being slave...." >> $LOGFILE 2>&1#sleep 10echo "backup Run SLAVEOF 10.0.11.2 cmd..." >> $LOGFILE$REDISCLI SLAVEOF 10.0.11.2 6379 >> $LOGFILE 2>&1--end--

/Opt/redis/sh/redis_fault.sh

--begin--#!/bin/bashLOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[fault]" >> $LOGFILEdate >> $LOGFILEsh /opt/redis/sh/redis_slave.sh--end--/opt/redis/sh/redis_stop.sh--begin--#!/bin/bashLOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[stop]" >> $LOGFILEdate >> $LOGFILE--end--

/Opt/redis/sh/redis_slave.sh

--begin--#!/bin/bashREDISCLI="/usr/local/bin/redis-cli"LOGFILE="/opt/redis/logs/keepalived-redis-state.log"echo "[slave]" >> $LOGFILEecho "Being slave...." >> $LOGFILE 2>&1sleep 35date >> $LOGFILEecho "slave Run SLAVEOF 10.0.11.2 cmd..." >> $LOGFILE$REDISCLI SLAVEOF 10.0.11.2 6379 >> $LOGFILE 2>&1--end--

Sleep 35 in redis_slave.sh indicates that, because redis on server A changes to the fault state after the service is started, and server A changes to the master state, if B is too fast to synchronize A (earlier than A's data synchronization from B), data will be lost. The purpose is to synchronize data from B first after A is started, and B is in slaveof,

Script Description: the logic of the script is that when the redis service on A and B is normal, A is the master, B is the slave
If A service is abnormal, B becomes the master. The "/usr/local/bin/redis-cli SLAVEOF NO ONE" command is used to disable data synchronization and change it to the Redis master.
If A switches back to the master after the service is started, the latest data is synchronized from B before it becomes the master. Execute "/usr/local/bin/redis-cli SLAVEOF 10.0.11.2 6379" on B"
Let B be the slave of A again, otherwise B is still the master.
In the redis master-slave architecture, you can use "/usr/local/bin/redis-cli-h 10.0.11.2 INFO" to view the current status. Check whether the current server is master or slave;


Under the command line
"Tail-30/opt/redis/logs/keepalived-redis-state.log" view the status transition of keepalived


"Tail-30/var/log/messages" to view changes in the keepalived virtual IP address.


The above has been tested in the actual production environment. Although some areas may not be reasonable, failover can be implemented and data will not be lost. Previously, you can switch the vip address based on the online tutorial, but there is a problem with the data.
Please let us know if there are better methods. Thank you!