Redis Sentinel mechanism and usage (II.)

https://segmentfault.com/a/1190000002685515

Overview of the Redis Sentinel mechanism and usage (i)

Redis-sentinel is a highly available (HA) solution that is officially recommended by Redis, and if Master is down, the Redis itself (including many of its clients) does not implement automatic primary and standby switching when using Redis for master-slave high-availability scenarios. The Redis-sentinel itself is also a standalone process that can monitor multiple master-slave clusters and discover that Master is down and able to switch on its own.

Its main function has the following points

Periodically monitor whether Redis works as expected;

If a Redis node is found to be running, it can notify another process (such as its client);

Ability to switch automatically. When a master node is unavailable, it is possible to elect one of the master's multiple slave (if there are more than one slave) as the new master, The other slave node changes the address of the master that it follows to the new address of the slave that is promoted to master.
<br/> configuration correction without failover

Sentinel will use the current configuration to set the master for monitoring, even if no failover are currently in progress. Especially:

Nodes that are confirmed as slaves according to the latest configuration claim to be master (refer to the network-isolated redis3 in the example above), and they will be reconfigured to the current master's slave.

If the slaves is connected to a wrong master, it will be corrected and connected to the correct master. slave elections and priorities

When a sentinel is ready to failover and receives the authorization from other Sentinel, it is necessary to elect a suitable slave to be the new master.

Slave's election will mainly assess the following aspects of slave:

Number of disconnects with master

Priority of Slave

Subscript for data replication (to evaluate how much master the slave currently has)

Process ID

If a slave loses contact with master more than 10 times, and each time it exceeds the configured maximum time to lose (down-after-milliseconds option), and if Sentinel finds slave missing when failover is in progress, Then this slave will be considered by Sentinel to be unsuitable for the new master.

A more rigorous definition is if a slave is continuously disconnected for longer than

(Down-after-milliseconds *) + milliseconds_since_master_is_in_sdown_state

Will be deemed to have lost their eligibility for election.
Slave that meet the above criteria will be listed in the Master candidate list and sorted according to the following order:

Sentinel first sorts according to the priority of the slaves, the smaller the priority ranking the higher the top (. ）。

If the priority is the same, look at the subscript of the copy, which one receives more copy data from Master, and which one is on top.

If the priority and subscript are the same, select the one with the smaller process ID.

A redis, either master or slave, must specify a slave priority in the configuration. Be aware that Master is also likely to become slave through failover.

If a Redis's slave priority is configured to 0, it will never be selected as master. But it will still replicate data from master. Sentinel and Redis authentication

When a master is configured to require a password to connect, both the client and slave need to provide a password when they connect.

Master sets its own password via Requirepass and does not provide a password to connect to this master.
Slave uses Masterauth to set the password to access master.

However, when Sentinel is used, because a master may become a slave, a slave may become master, so both of these configuration items need to be set at the same time. Sentinel API

Sentinel is running on port 26379 by default and Sentinel supports the Redis protocol, so you can use the REDIS-CLI client or other available clients to communicate with Sentinel.

There are two ways to communicate with Sentinel:

One is to send a message to it directly using the client

Another is to use the Publish/subscribe model to subscribe to sentinel events, such as failover, or a Redis instance running in error, and so on. Sentinel Command

The legitimate commands supported by Sentinel are as follows:

PING Sentinel reply Pong.

SENTINEL Masters Displays all the master that is monitored and their status.

SENTINEL Master <master name> Displays the information and status of the specified master;

SENTINEL Slaves <master name> displays all slave of the specified master and their status;

SENTINEL get-master-addr-by-name <master name> returns the IP and port of the specified master, if failover is in progress or failover is completed, The IP and port of the slave that are promoted to master will be displayed.

SENTINEL reset <pattern> Reset name matches all of the master's state information for the regular expression, clear its previous status information, and slaves information.

Sentinel Failover <master name> enforces sentinel execution failover and does not need to be approved by other Sentinel. However, the latest configuration will be sent to other Sentinel after failover. dynamically modifying Sentinel configuration

Starting with redis2.8.4, Sentinel provides a set of APIs for adding, deleting, and modifying master configurations.

It is important to note that if you modify a sentinel configuration through the API, Sentinel does not tell the modified configuration to other Sentinel. You need to manually send a modified configuration command to multiple Sentinel.

Here are some commands to modify Sentinel configuration:

Sentinel Monitor <name> <ip> <port> <quorum> This command tells Sentinel to listen to a new master

Sentinel REMOVE <name> Command Sentinel abandons monitoring of a master

SENTINEL Set <name> <option> <value> This command is much like the Config SET command of Redis, which is used to change the configuration of the specified master. Supports multiple <option><value>. Examples include the following:

SENTINEL SET objects-cache-master Down-after-milliseconds 1000

As long as the configuration item exists in the configuration file, it can be set with the Sentinel set command. This can also be used to set the properties of master, such as quorum (votes), without having to delete master first and then re-add master. For example:

SENTINEL SET Objects-cache-master Quorum 5

Add or Remove Sentinel

With Sentinel Auto-discovery, it's easy to add a sentinel to your cluster, all you need to do is monitor to a master, The newly added Sentinel will then be able to obtain information from other Sentinel and Masterd all slave.

If you need to add more than one sentinel, it is recommended that you add one after another to prevent network isolation from causing problems. You can add a sentinel every 30 seconds. Finally, you can use Sentinel Master Mastername to check if all sentinel has been monitored by master.

Deleting a Sentinel is a bit tricky: Because Sentinel never deletes an already existing sentinel, even if it has been out of touch with the Organization for a long time.
To delete a sentinel, you should follow these steps:

Stop the Sentinel that you want to delete

Send a Sentinel Reset * command to all other Sentinel instances, if you want to reset the sentinel above the specified master, just change the * number to a specific name, and note that it needs to be sent one after the other, each sending at a distance of no less than 30 seconds.

Check to see if all Sentinels have a consistent current sentinel count. Use Sentinel MASTER mastername to query. Delete old master or unreachable slave

Sentinel will always record good one master's slaves, even if slave has been out of the Organization for a long time. This is useful because the Sentinel cluster must have the ability to reconfigure a recovery-usable slave.

Also, after failover, the failed master will be marked as a slave of the new master, so that when it becomes available, the data will be copied from new master.

Then, sometimes you want to permanently delete a Slave (possibly it used to be a master), you just send a sentinel RESET Master command to all the Sentinels, They will update the slave in the list to copy the master data correctly. Publish/Subscribe

A client can send a command to a sentinel to subscribe to events for a channel, and Sentinel notifies all subscribed clients when a specific event occurs. It is important to note that the client can only subscribe and cannot be published.

The name of the subscribed channel is the same as the name of the event. For example, a channel named Sdown will publish all Sdown-related messages to subscribers.

If you want to subscribe to all messages, simply use Psubscribe *

Here is the message format for all the messages you can receive, if you subscribe to all the messages. The first word is the name of the channel, and the other is the format of the data.

Note: The following format for instance details is:

<instance-type> <name> <ip> <port> @ <master-name> <master-ip> <master-port >

If the Redis instance is a master, then messages after @ will not be displayed.

    +reset-master <instance details>--When master is reset.
    +slave <instance details>-When a slave is detected and added into the slave list. +failover-state-reconf-slaves <instance details>--reconf-slaves < When failover status changes to +failover-detected state Instance details>--when failover occurs +slave-reconf-sent <instance details>--Sentinel sends slaveof command to reconfigure it +SL
    Ave-reconf-inprog <instance details>--Slave was reconfigured to another master slave, but data replication did not occur.
    +slave-reconf-done <instance details>--Slave is reconfigured for another master slave and data replication is already in sync with master.
    -dup-sentinel <instance details>--This event may occur when a sentinel is restarted when the redundant sentinel on the specified master is deleted.
    +sentinel <instance details>-When Master adds a sentinel.
    +sdown <instance details>--when entering the Sdown state;
    -sdown <instance details>--when leaving the Sdown state.
    +odown <instance details>--when entering the Odown state.
    -odown <instance details>--when leaving the Odown state.
    +new-epoch <instance details>-When the current configuration version is updated. +try-failover <instance details>-to meet the failover conditions, awaiting other sentinel elections.
    +elected-leader <instance details>--was elected to carry out the failover.
    +failover-state-select-slave <instance details>--When you start to select a slave when you elect a new master. No-good-slave <instance details>--no suitable slave to serve as the new master Selected-slave <instance details>--found a suitable Slav
    E to serve as the new master Failover-state-send-slaveof-noone <instance details>-When switching the identity of the slave chosen for the new master.
    Failover-end-for-timeout <instance details>--failover failed due to timeout.
    Failover-end <instance details>--when failover was successfully completed. Switch-master <master name> <oldip> <oldport> <newip> <newport>-When Master's address is changed.
    Usually this is the message that the client is most interested in.
    +tilt--Enter tilt mode. -tilt--Exit tilt mode.

TILT Mode

Redis Sentinel relies heavily on system time, for example, it uses the system time to determine how long a ping reply takes.
However, if the system time has been modified, or if the system is busy, or if the process is blocked, Sentinel may be running abnormally.
When the stability of the system decreases, the tilt mode is a kind of protection mode that Sentinel can enter. When entering tilt mode, Sentinel will continue to monitor the work, but it will not have any other action, and it will not respond to commands like IS-MASTER-DOWN-BY-ADDR, because it is in tilt mode, the ability to detect failed nodes has become untrustworthy.
If the system returns to normal for 30 seconds, Sentinel exits titl mode.