5. Redis Monitoring Tool--redis Sentinel usage instructions and precautions

With the previous 4 notes, you have already learned about the basic concept and configuration of Redis, and this note highlights how to monitor the operation of Redis using the official Redis Sentinel tool. In addition, some considerations for sentinel use are discussed.

1. Redis Sentinel features
        Redis Sentinel is a set of distributed systems for managing Redis instances, with 3 main tasks to complete:
         1) monitoring: Continuously monitor whether Redis master or slave instances are running as expected
         2) Notification: If the monitored Redis instance is running abnormally, Sentinel notifies the outside world (person or program) via API
3) Automation failover: If the master instance fails, Sentinel will re-elect the master and initiate automatic failover: Select the slave instance of the slave-priority minimum and promote it to master, while modifying the other SLA VE is configured so that its master configuration item points to the new master, and when old master resumes restarting, it is automatically demoted to the slave of new master. Finally, depending on the configuration, Redis Sentinel will also notify the new master address to the application that is currently accessing Redis.

2. Redis Sentinel Deployment
Sentinel, as a distributed system tool, recommends multiple machine room deployments .

2.1 Sentinel configuration file
each Sentinel instance has 6 configuration items, which are configured according to the actual deployment of the Redis cluster, as shown in the following example:

[Plain] View plain copy 650) this.width=650; "Src=" https://code.csdn.net/assets/ Code_ico.png "alt=" on code to view the "style=" Border:none; "height=""width=" [/>] 650 this.width=650; "Src=" Https://code.csdn.net/assets/ico_fork.svg "alt=" derives to My Code slice "style=" border:none; "height=" width=

2. Port 26329

4. Sentinel Monitor MyMaster 127.0.0.1 6379 2

6. Sentinel Down-after-milliseconds MyMaster 60000

8. Sentinel Failover-timeout MyMaster 180000

10. Sentinel Parallel-syncs MyMaster

12. Sentinel Notification-script <master-name> <script-path>

which
PORT: Specifies the listening port for Sentinel (that is, the port on which the TCP connection is established with Redis server or client)
Monitor: Specifies the Redis instance for Sentinel to monitor, including the alias of a Redis instance and the Ip+port of the Redis instance, the last number 2 of the line indicatesat least2 Setinel instances when a Redis server exception is detected, the status of Redis server is judged as real fail. That is, if the configuration is 2, but Sentinel deploys only 1 units in the actual deployment, Sentinel will not give any warning even if the Redis instance is already dead.this requires special attention.
Down-after-milliseconds: Specifies how long the sentinel monitoring to the Redis instance will continue to be an exception, and the state is judged to be down.if the actual business requires Sentinel to adjudicate a Redis instance exception as soon as possible, the value can be suitably small.
Failover-timeout: If Sentinel fails to complete the failover operation within this configuration value (i.e., Master/slave automatically switch on failure), this failover is considered to be unsuccessful. There are 4 uses for this configuration, which can be referred to in sentinel.conf and is limited to space, which is not described here.
Parallel-syncs: Specifies the maximum number of slave instances that are simultaneously reconfigure by Sentinel during the failover process. Due to the reconfigure process, the corresponding slave will interrupt the response to the client request, so to avoid all slave at the same time, the value needs to be suitably small.
Notification-script: Specifies the alert script invoked when Sentinel detects an instance exception that Master-name points to. This configuration item is optional, but the online system recommends configuration.

2.2 Boot monitoring System
        When the configuration file is modified, start each monitoring process, for example:

[Plain] View plain copy 650) this.width=650; "Src=" https://code.csdn.net/assets/ Code_ico.png "alt=" on code to view the "style=" Border:none; "height=""width=" [/>] 650 this.width=650; "Src=" Https://code.csdn.net/assets/ico_fork.svg "alt=" derives to My Code slice "style=" border:none; "height=" width=

1. Nohup./bin/redis-sentinel./conf/sentinel.conf >./log/redis-sentinel.log 2>&1 &

2.3 Sentinel Use scene measurement
To investigate and master Sentinel usage, I built the Redis test environment and made a series of experiments,Detailed Description。
Special Note: As the following content may be related to the company intranet address, so as to avoid unnecessary trouble, the text or the presence of IP address where the smear, but does not affect the description of the problem.
Experimental environment (one Master/two Slaves/two Sentinels):
A. A master (slave-priority of 100) is deployed on IP as xx.xx.234.67 on the machine;
B. Two slaves (slave-priority 90/100 respectively) are deployed inIP is xx.xx.234.49 on the machine;
C. Enabling two Sentinel processes to monitor the status of Redis clusters
6 case tests were done and the results were as follows:
 Case1: After starting the master process and 2 slave processes in turn, and then starting 2 Sentinel processes, Sentinel can identify the master-slave relationship normally
 Case2: With the shutdown command to stop master, the Sentinel automatically selected the Slave-priority small slave process as new master, and automatically points the master of another slave process to the new master
 CASE3: On CASE2 basis, restarting the old Master,sentinel will downgrade it to Slave, whose master points to the new master selected by Case2
 Case4: Match the slave-priority of master and 2 slave instances to different values, shutdown the current master on Case1 basis, After Sentinel has chosen a new master and reconfigure other instances to point them to the new master (from the old Master exception to the time that the Sentinel re-select Master was triggered by the user specified by the sentinel.conf down-after-milliseconds configuration item), restart the old master, the final state of the system is consistent with CASE3, that is, the old Master has been downgraded to slave, and its master points to the new master chosen by Sentinel.If the old master is restarted before Sentinel has selected a new master but has not completed another instance of reconfigure, the entire system will have an exception that cannot be selected for new master, as described in Case5 below.
 CASE5: Match the slave-priority of master and 2 slave instances to the same values, shutdown the current master on Case1 basis, and after Sentinel has chosen the new master and reconfigure other instances to point them to the new master, Restart Old Master, the final state of the system is consistent with CASE3, that is, the old master has been downgraded to slave, and its master points to the new master chosen by Sentinel. In cases where all slave-priority are configured with the same value, Sentinel promotes the Runid smallest slave in each slave instance to master (provided it is redis.conf to master in the slave corresponding promote).similar to the anomalies in Case4, if you restart Old master before Sentinel selects the new master but has not yet completed the reconfigure of other instances, you will find Sentinel's automatic failover mechanism is already messy.
Detailed exception conditions are described below.
When the old master is deployed on a machine with IP xx.xx.234.67 and port defaults to a 6379,sentinel toggle exception, the output of the info command for that old Master is as follows:
650) this.width=650; "Src=" http://img.blog.csdn.net/20131230170014875?watermark/2/text/ ahr0cdovl2jsb2cuy3nkbi5uzxqvc2x2agvy/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/dissolve/70/gravity/ Southeast "style=" Border:none; "/>
slave-00 instances inThe IP is xx.xx.234.49 on the machineWith Port 6378,after Sentinel switch exception,The info output is as follows:
650) this.width=650; "Src=" http://img.blog.csdn.net/20131230170129234?watermark/2/text/ ahr0cdovl2jsb2cuy3nkbi5uzxqvc2x2agvy/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/dissolve/70/gravity/ Southeast "style=" Border:none; "/>
SLAVE-01 instances inmachines with IP xx.xx.234.49On(same-machine deployment with slave-00)and the port with the 6377,info output is as follows:
650) this.width=650; "Src=" http://img.blog.csdn.net/20131230170200375?watermark/2/text/ ahr0cdovl2jsb2cuy3nkbi5uzxqvc2x2agvy/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/dissolve/70/gravity/ Southeast "style=" Border:none; "/>
From the output of the above 3 Redis instances, 3 think they are slave,the entire system has no master！ Among them, located in xx.xx.234.67Old Master (note the Master_host field of the 1th figure above) and the salve-00 in xx.xx.234.49 (Notice the Master_host field in the 2nd figure aboveare considered slave-01 as new master, while the slave-01 in xx.xx.234.49 thinks he is still slave, and that old master is still master ( Note the Master_host field of the 3rd figure above )
        from the Sentinel process log, it is not possible to elect a new master, which means Sentinel cannot confirm which of the two master candidates is the new master and switches between two instances frequently:
650) this.width=650; "Src=" http://img.blog.csdn.net/20131230172424312?watermark/2/text/ ahr0cdovl2jsb2cuy3nkbi5uzxqvc2x2agvy/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/dissolve/70/gravity/ Southeast "style=" Border:none; "/>
                 Case 6:after the system has entered the abnormal state shown in Case5, shutdown two master candidates one instance, Sentinel still can not select the main, until 3 instances all shutdown, the entire system is still no master. It is essential to determine that the internal logic state of the monitoring system is chaotic.

2.4 Conclusion
If the master instance fails, it is best to wait for Sentinel to elect new master and stabilize (choose the newly master and complete the switchover time and configuration, the typical value within 1 minutes), and then restart the old master, to avoid triggering Sentinel's miscarriage, Causes the entire system to fail to elect new master.

This article is from "Linux OPS rookie" blog, please be sure to keep this source http://zhangchuang.blog.51cto.com/10884116/1786973

5. Redis monitoring Tool--redis Sentinel usage instructions and precautions