Redis Sentinel mechanism and usage (II.)

Overview

Redis-sentinel is a highly available (HA) solution that is officially recommended by Redis, and if Master is down, the Redis itself (including many of its clients) does not implement automatic primary and standby switching when using Redis for master-slave high-availability scenarios. The Redis-sentinel itself is also a standalone process that can monitor multiple master-slave clusters and discover that Master is down and able to switch on its own.

Its main function has the following points

• Periodically monitor whether Redis works as expected;

• If a Redis node is found to be running, it can notify another process (such as its client);

• Ability to switch automatically. When a master node is unavailable, it is possible to elect one of the master's multiple slave (if there are more than one slave) as the new master, The other slave node changes the address of the master that it follows to the new address of the slave that is promoted to master.
  <br/>

Configuration correction without failover

Sentinel will use the current configuration to set the master for monitoring, even if no failover are currently in progress. Especially:

• Nodes that are confirmed as slaves according to the latest configuration claim to be master (refer to the network-isolated redis3 in the example above), and they will be reconfigured to the current master's slave.

• If the slaves is connected to a wrong master, it will be corrected and connected to the correct master.

Slave elections and priorities

When a sentinel is ready to failover and receives the authorization from other Sentinel, it is necessary to elect a suitable slave to be the new master.

Slave's election will mainly assess the following aspects of slave:

• Number of disconnects with master

• Priority of Slave

• Subscript for data replication (to evaluate how much master the slave currently has)

• Process ID

If a slave loses contact with master more than 10 times, and each time it exceeds the configured maximum time to lose ( down-after-milliseconds option ), and if Sentinel discovers slave mismatch when failover is in progress, Then this slave will be considered by Sentinel to be unsuitable for the new master.

A more rigorous definition is if a slave is continuously disconnected for longer than

(Down-after-milliseconds *) + milliseconds_since_master_is_in_sdown_state

Will be deemed to have lost their eligibility for election.
Slave that meet the above criteria will be listed in the Master candidate list and sorted according to the following order:

1. Sentinel first sorts according to the priority of the slaves, the smaller the priority rank the higher the top (? ）。

2. If the priority is the same, look at the subscript of the copy, which one receives more copy data from Master, and which one is on top.

3. If the priority and subscript are the same, select the one with the smaller process ID.

A redis, either master or slave, must specify a slave priority in the configuration. Be aware that Master is also likely to become slave through failover.

If a Redis's slave priority is configured to 0, it will never be selected as master. But it will still replicate data from master.

Sentinel and Redis Authentication

When a master is configured to require a password to connect, both the client and slave need to provide a password when they connect.

Master requirepass does not provide a password to connect to this master by setting its own password.
Slave pass masterauth to set the password to access master.

However, when Sentinel is used, because a master may become a slave, a slave may become master, so both of these configuration items need to be set at the same time.

Sentinel API

Sentinel is running on port 26379 by default and Sentinel supports the Redis protocol, so you can use the REDIS-CLI client or other available clients to communicate with Sentinel.

There are two ways to communicate with Sentinel:

• One is to send a message to it directly using the client

• Another is to use the Publish/Subscribe model to subscribe to sentinel events, such as failover, or a Redis instance running in error, and so on.

Sentinel command

The legitimate commands supported by Sentinel are as follows:

• PINGSentinel replies PONG .

• SENTINEL mastersDisplays all the master that is monitored and their status.

• SENTINEL master <master name>Displays the information and status of the specified master;

• SENTINEL slaves <master name>Displays all slave of the specified master and their status;

• SENTINEL get-master-addr-by-name <master name>Returns the IP and port of the specified master, and if failover is in progress or failover is completed, the IP and port of the slave promoted to master will be displayed.

• SENTINEL reset <pattern>Resets the name to match all of the master's state information for the regular expression, clear its previous status information, and slaves information.

• SENTINEL failover <master name>Enforces sentinel execution failover and does not need to be approved by other Sentinel. However, the latest configuration will be sent to other Sentinel after failover.

Dynamically modifying Sentinel Configuration

Starting with redis2.8.4, Sentinel provides a set of APIs for adding, deleting, and modifying master configurations.

It is important to note that if you modify a sentinel configuration through the API, Sentinel does not tell the modified configuration to other Sentinel. You need to manually send a modified configuration command to multiple Sentinel.

here are some commands to modify Sentinel configuration: Sentinel MONITOR<name> <IP> <Port> <Quorum>This command tells Sentinel to listen to a new Master Sentinel REMOVE.<name>Command Sentinel to abort monitoring Sentinel SET for a master<name> <option> <value>This command is much like the config set command of Redis, which is used to change the configuration of the specified master. Supports multiple<option><value>. For example, the following instance: Sentinel set Objects-cache-master down-after-milliseconds 1000 can be set with the Sentinel set command only if the configuration item exists in the configuration file. This can also be used to set the properties of master, such as quorum (votes), without having to delete master first and then re-add master. For example:

SENTINEL SET Objects-cache-master Quorum 5

Add or Remove Sentinel

With Sentinel Auto-discovery, it's easy to add a sentinel to your cluster, all you need to do is monitor to a master, The newly added Sentinel will then be able to obtain information from other Sentinel and Masterd all slave.

If you need to add more than one sentinel, it is recommended that you add one after another to prevent network isolation from causing problems. You can add a sentinel every 30 seconds. Finally, you can SENTINEL MASTER mastername check to see if all sentinel has been monitored by master.

Deleting a Sentinel is a bit tricky: Because Sentinel never deletes an already existing sentinel, even if it has been out of touch with the Organization for a long time.
To delete a sentinel, you should follow these steps:

1. Stop the Sentinel that you want to delete

2. Send a SENTINEL RESET * command to all other Sentinel instances, if you want to reset the sentinel above the specified master, just change the * number to a specific name, note that it needs to be sent one after the other, and each time the interval is not less than 30 seconds.

3. Check to see if all Sentinels have a consistent current sentinel count. Use SENTINEL MASTER mastername to query.

Delete old Master or unreachable slave

Sentinel will always record good one master's slaves, even if slave has been out of the Organization for a long time. This is useful because the Sentinel cluster must have the ability to reconfigure a recovery-usable slave.

Also, after failover, the failed master will be marked as a slave of the new master, so that when it becomes available, the data will be copied from new master.

Then, sometimes you want to permanently delete a slave (it might have been a master), you just send a SENTINEL RESET master command to all the Sentinels, and they will update the slave in the list to copy the master data correctly.

Publish/Subscribe

A client can send a command to a sentinel to subscribe to events for a channel, and Sentinel notifies all subscribed clients when a specific event occurs. It is important to note that the client can only subscribe and cannot be published.

The name of the subscribed channel is the same as the name of the event. For example, a channel named Sdown will publish all Sdown-related messages to subscribers.

If you want to subscribe to all messages, simply use thePSUBSCRIBE *

Here is the message format for all the messages you can receive, if you subscribe to all the messages. The first word is the name of the channel, and the other is the format of the data.

Note: The following format for instance details is:

<instance-type> <name> <ip> <port> @ <master-name> <master-ip> <master-port >

If the Redis instance is a master, then messages after @ will not be displayed.

+reset-master<instanceDetails>--When master is reset. +slave<instanceDetails>--When a slave is detected and added into the slave list. +failover-state-reconf-slaves<instanceDetails>--When the failover state becomes reconf-slaves state +failover-detected<instanceDetails>--When failover occurs +slave-reconf-sent<instanceDetails>--Sentinel sends slaveof command to reconfigure it +slave-reconf-inprog<instanceDetails>--Slave is reconfigured to another master's slave, but data replication has not occurred. +slave-reconf-done<instanceDetails>--Slave is reconfigured to another master's slave and data replication is already in sync with master. -dup-sentinel<instanceDetails>--When redundant sentinel on the specified master is deleted (this event may occur when a Sentinel restarts). +sentinel<instanceDetails>-When Master adds a sentinel. +sdown<instanceDetails>--When entering the Sdown state; -sdown<instanceDetails>--When leaving the Sdown state. +odown<instanceDetails>--When entering the Odown state. -odown<instanceDetails>--When leaving the Odown state. +new-epoch<instanceDetails>--When the current configuration version is updated. +try-failover<instanceDetails>-meet failover conditions and await other sentinel elections. +elected-leader<instanceDetails>-When elected to carry out the failover. +failover-state-select-slave<instanceDetails>--start to select a slave when the new master is elected. No-good-slave<instanceDetails>-no suitable slave to serve as new master Selected-slave<instanceDetails>--found a suitable slave to serve as the new master Failover-state-send-slaveof-noone<instanceDetails>-When switching the identity of the slave selected as the new master. Failover-end-for-timeout<instanceDetails>--Failover failed due to timeout. Failover-end<instanceDetails>--When failover is successfully completed. Switch-master<Mastername> <Oldip> <Oldport> <Newip> <Newport>--When the address of master is changed.    Usually this is the message that the client is most interested in.    +tilt--Enter tilt mode. -tilt--Exit tilt mode. 

TILT mode

Redis Sentinel relies heavily on system time, for example, it uses the system time to determine how long a ping reply takes.
However, if the system time has been modified, or if the system is busy, or if the process is blocked, Sentinel may be running abnormally.
When the stability of the system decreases, the tilt mode is a kind of protection mode that Sentinel can enter. When entering tilt mode, Sentinel will continue to monitor the work, but it will not have any other action, it will not respond to is-master-down-by-addr such commands, because it is in the tilt mode, the ability to detect the failure node has become untrustworthy.
If the system returns to normal for 30 seconds, Sentinel exits titl mode.

-busy status

Note: This feature is not yet implemented.

When a script runs longer than the configured run time, Sentinel returns a-busy error signal. If this happens before triggering a failover, Sentinel will send a script kill command, if script is read-only, it will execute successfully.

Published on April 17, 2015

1190000002685515

Redis Sentinel mechanism and usage (II.)