Sentinel-based Redis high-availability scenario

First, Redis Sentinel is a Redis-enabled cluster management tool with the main features

· Monitoring (Monitoring): Redis Sentinel monitors the primary server and the running state from the server in real time.

· Reminder (Notification): When a problem occurs with a Redis server that is being monitored, redis Sentinel can send notifications to the system administrator, or it can send notifications to other programs through the API.

· Automatic failover (Automatic failover): When a primary server does not work, Redis Sentinel can upgrade one from the server to the primary server and configure the other slave servers to use the new primary server. When an application connects to a Redis server, Redis Sentinel reports the new primary server address and port.

Redis Sentinel is a distributed system that allows you to run multiple Sentinel processes in a schema that communicate with each other to determine if a primary server is disconnected and if failover should be performed.

When you configure Redis Sentinel, you need at least 1 master and one slave. When Master fails, Redis Sentinel reports a failure warning, promotes slave to master with automatic failover, and provides read-write services, and when the failed master resumes, Redis Sentinel automatically recognizes it. Automatically converts master to slave and completes data synchronization.

Redis Sentinel enables Redis 0 manual intervention and m-s switching within a short period of time, reducing business impact time.

Second, the topological structure

Redis and Redis Sentinel are deployed in two servers, respectively. When Redis in master fails (Redis process terminates, server zombies, server power outages, and so on), Redis Sentinel switches Master permissions to slave Redis and changes the read-only mode to readable writable mode. The application uses Redis sentinal to determine the current master Redis location for reconnection.

According to the business model, two kinds of topologies can be developed: single M-S structure and double m-s structure. If you have enough servers, you can configure a multi-m-s structure.

1. Single M-S structure

The single m-s structure is characterized by the configuration of Master Redis (redis-1m) and Master Sentinel (sentinel-1m) in the master server. Slave Redis (Redis-1s) and slave Sentinel (SENTINEL-1S) are configured in the slave server. Where Master Redis can provide read and write services, but slave Redis can only provide read-only services. Therefore, you can choose to place the read-only business in slave redis in the case of a larger business pressure.

2. Double M-S structure

The dual m-s structure is characterized by the configuration of a master Redis on each server and the deployment of a slave redis. Two Redis Sentinel monitors 4 Redis at the same time. Two master Redis can provide read and write services to the application at the same time, even if one of the servers fails, and another server can run two master Redis to provide read and write services. The disadvantage is that data sharing is not possible between two master Redis and is not suitable for applications with large numbers of user data associations.

3. Comparison of pros and cons

Each of the two structures has advantages and disadvantages, and is suitable for different application scenarios:

The single m-s structure is applicable to different user data, but the application can realize the business model of read/write separation. Master mainly provides write operation, slave mainly provides read operation, make full use of hardware resources.

The dual (multi) m-s structure is suitable for non-existent or less data-related business models, and read-write efficiency is two (multiple) times of single m-s, but requires a single server to assume two Mater Redis resource requirements when a fault occurs.

Third, configure the deployment

The single m-s structure and the dual m-s configuration are similar, the following is an example of a dual m-s structure configuration.

1. Redis Configuration

1) Master Redis configuration

Configure redis-1m on the server-1m

# VI Redis-1m.conf

# # Master Redis-1m

# # Daemonize Default to No, modify to Yes to enable background run

Daemonize Yes

# Redis Default PID file location Redis.pid

When #当运行多个 Redis service, you need to specify a different PID file and port

Pidfile Redis-1m.pid

# #端口号

Port 6379

# #验证口令

Requirepass *************

Masterauth *************

#绑定可连接Redis的IP地址, do not set will process all requests

# bind 127.0.0.1

#客户端连接的超时时间, in seconds, closes the connection after timeout (0 is not set)

Timeout 0

#日志记录等级

LogLevel Notice

#设置数据库的个数

Databases 16

#日志刷新策略 (Master Disabled)

#save 900 1

#save 300 10

#save 60 10000

#是否使用压缩镜像备份

Rdbcompression Yes

#镜像备份文件的文件名

Dbfilename Redis-1m_dump.rdb

#镜像备份路径, the default value is./

Dir/redis/backup

#设置该数据库为其他数据库的从数据库, the main library does not need to be set

#slaveof

# slaveof

#指定与主数据库连接时需要的密码验证, the main library does not need to be set

#masterauth

#masterauth

#如果 Slave-serve-stale-data is set to ' no ', slave returns the "SYNC with Master in #progress" error message, except for the info and slaveof commands.

Slave-serve-stale-data Yes

#客户端连接访问口令

# Requirepass Foobared

#限制同时连接的客户数量 to prevent excessive client memory exhaustion. If there is enough memory to not set the #

#maxclients 10000

#设置redis能够使用的最大内存.

# MaxMemory

# #启用增量 (Master disabled)

AppendOnly No

#增量日志文件名, the default value is Appendonly.aof

Appendfilename appendonly.aof

How often #设置对 appendonly.aof files are synchronized

#always indicates that each write operation is synchronized, and Everysec indicates that the write operation is cumulative and synchronized once per second.

#no表示等操作系统进行数据缓存同步到磁盘, all synchronized, everysec indicates that the write operation is tired # product, synchronized once per second

Appendfsync everysec

#是否重置Hash表

#设置成yes后redis将每100毫秒使用1毫秒CPU时间来对redis的hash表重新hash, # #可降低内存的使用. When the use of the scene has a more stringent real-time requirements, can not accept Redis occasionally to please # #求有2毫秒的延迟的话, this is configured to No. Without such strict real-time requirements, you can set the #yes to free up memory as quickly as possible.

activerehashing Yes

# #Slave开启只读模式

Slave-read-only Yes

Configure redis-2m on the Server-1s

# VI Redis-2m.conf

# # Master Redis-2m

# Redis Default PID file location Redis.pid

When #当运行多个 Redis service, you need to specify a different PID file and port

Pidfile Redis-2m.pid

#镜像备份文件的文件名

Dbfilename Redis-1m_dump.rdb

#日志刷新策略 (slave enabled)

Save 900 1

Save 300 10

Save 60 10000

#-----------------Other parameters remain consistent with redis-1m-----------------

Daemonize Yes

#......

2) Slave Redis configuration

Configure Redis-1s on the Server-1s

# Redis Default PID file location Redis.pid

When #当运行多个 Redis service, you need to specify a different PID file and port

Pidfile Redis-1s.pid

# #端口号

Port 7379

#镜像备份文件的文件名

Dbfilename Redis-1s_dump.rdb

#设置该数据库为其他数据库的从数据库, the main library does not need to be set

Slaveof server-1m 6379

# #启用增量 (Master disabled)

AppendOnly Yes

#-----------------Other parameters remain consistent with redis-1m-----------------

Daemonize Yes

#......

Configure Redis-2s on the server-1m

# Redis Default PID file location Redis.pid

When #当运行多个 Redis service, you need to specify a different PID file and port

Pidfile Redis-2s.pid

# #端口号

Port 7379

#镜像备份文件的文件名

Dbfilename Redis-2s_dump.rdb

#设置该数据库为其他数据库的从数据库, the main library does not need to be set

Slaveof Server-1s 6379

#-----------------Other parameters remain consistent with redis-2m-----------------

Daemonize Yes

#......

2. Redis Sentinel Configuration

Configure sentinel-1m on the server-1m

VI sentinel-1m.conf

#Master redis-1m

#hostname server-1m

#ip 192.168.84.129

#端口号

Port 26379

Sentinel Monitor server-1m 192.168.84.129 6379 1

Sentinel Down-after-milliseconds server-1m 5000

Sentinel Failover-timeout server-1m 900000

Sentinel Can-failover server-1m Yes

Sentinel Parallel-syncs server-1m 1

#Master Redis-1s

#hostname Server-1s

#ip 192.168.84.128

Sentinel Monitor Server-1s 192.168.84.128 6379 1

Sentinel Down-after-milliseconds Server-1s 5000

Sentinel Failover-timeout Server-1s 900000

Sentinel Can-failover Server-1s Yes

Sentinel Parallel-syncs Server-1s 1

Configure Sentinel-1s on the Server-1s

#Master redis-1m

#hostname server-1m

#ip 192.168.84.128

#端口号

Port 27379

Sentinel Monitor server-1m 192.168.84.129 6379 1

Sentinel Down-after-milliseconds server-1m 5000

Sentinel Failover-timeout server-1m 900000

Sentinel Can-failover server-1m Yes

Sentinel Parallel-syncs server-1m 1

#Master Redis-1s

#hostname Server-1s

#ip 192.168.84.128

Sentinel Monitor Server-1s 192.168.84.128 6379 1

Sentinel Down-after-milliseconds Server-1s 5000

Sentinel Failover-timeout Server-1s 900000

Sentinel Can-failover Server-1s Yes

Sentinel Parallel-syncs Server-1s 1

3. Start the service

Server-1m starting Redis

Redis-server redis-1m.conf

Redis-server redis-2s.conf

Server-1s starting Redis

Redis-server redis-1s.conf

Redis-server redis-2m.conf

Detect synchronization

Master Log Check:

[28162] 23:32:11.810 * DB loaded from append only file:0.000 seconds

[28162] 23:32:11.810 * The server is now a ready-to-accept connections on port 6379

[28162] 23:32:35.360 * Slave ask for synchronization

[28162] 23:32:35.360 * Starting BGSAVE for SYNC

[28162] 23:32:35.361 * Background saving started by PID 28170

[28170] 23:32:35.373 * DB saved on disk

[28170] 23:32:35.375 * rdb:0 MB of memory used by Copy-on-write

[28162] 23:32:35.437 * Background saving terminated with success

[28162] 23:32:35.438 * Synchronization with Slave succeeded

Slave log Check:

[28091] 23:27:15.908 * DB loaded from append only file:0.001 seconds

[28091] 23:27:15.908 * The server is now a ready-to-accept connections on port 6389

[28091] 23:27:15.944 * Connecting to MASTER ...

[28091] 23:27:15.947 * MASTER <-> SLAVE Sync started

[28091] 23:27:15.951 * Non blocking Connect for SYNC fired the event.

[28091] 23:27:15.952 * Master replied to PING, replication can continue ...

[28091] 23:27:15.990 * Master <-> SLAVE sync:receiving bytes from master

[28091] 23:27:15.990 * MASTER <-> SLAVE sync:loading DB in memory

[28091] 23:27:15.991 * MASTER <-> SLAVE sync:finished with success

[28091] 23:27:15.993 * Background Append only file rewriting started by PID 28094

[28094] 23:27:15.998 * SYNC append only file rewrite performed

[28094] 23:27:15.998 * AOF rewrite:0 MB of memory used by Copy-on-write

[28091] 23:27:16.047 * Background AOF rewrite terminated with success

[28091] 23:27:16.047 * Parent diff successfully flushed to the rewritten AOF (0 bytes)

[28091] 23:27:16.048 * Background AOF rewrite finished successfully

Launch Sentinel Service

Redis-server sentinel-1m.conf--sentinel

[28156] 23:33:22.337 * +slave slave 192.168.84.129:6389 192.168.84.129 6389 @server -1s 192.168.84.128 6379

[28156] 23:39:37.071 # +sdown Sentinel 192.168.84.128:26389 192.168.84.128 26389 @server -1s 192.168.84.128 6379

[28156] 23:39:37.072 # +sdown Sentinel 192.168.84.128:26389 192.168.84.128 26389 @server -1m 192.168.84.129 6379

[28156] 23:40:05.438 #-sdown Sentinel 192.168.84.128:26389 192.168.84.128 26389 @server -1s 192.168.84.128 6379

[28156] 23:40:05.438 #-sdown Sentinel 192.168.84.128:26389 192.168.84.128 26389 @server -1m 192.168.84.129 6379

[28156] 23:40:10.262 *-dup-sentinel masterserver-1s 192.168.84.128 6379 #duplicate of 192.168.84.128:26389 or e453 a45a5e1421519dcbe00a199f203579b27b0f

[28156] 23:40:10.263 * +sentinel Sentinel 192.168.84.128:26389 192.168.84.128 26389 @server -1s 192.168.84.128 6379

[28156] 23:40:10.263 *-dup-sentinel masterserver-1m 192.168.84.129 6379 #duplicate of 192.168.84.128:26389 or e453 a45a5e1421519dcbe00a199f203579b27b0f

[28156] 23:40:10.263 * +sentinel Sentinel 192.168.84.128:26389 192.168.84.128 26389 @server -1m 192.168.84.129 6379

Redis-server sentinel-1s.conf--sentinel

[16383] 23:40:05.499 * +slave slave 192.168.84.129:6389 192.168.84.129 6389 @server -1s 192.168.84.128 6379

[16383] 23:40:05.500 * +slave slave 192.168.84.128:6389 192.168.84.128 6389 @server -1m 192.168.84.129 6379

[16383] 23:40:06.098 * +sentinel Sentinel 192.168.84.129:26379 192.168.84.129 26379 @server -1s 192.168.84.128 6379

[16383] 23:40:08.404 * +sentinel Sentinel 192.168.84.129:26379 192.168.84.129 26379 @server -1m 192.168.84.129 6379

4, fault simulation detection

Analog redis-1m (Master 192.168.84.129 6379) failure, Sentinel automatically detects the status, then switches to Redis-1s (Slave 192.168.84.128 6389), The redis-1m is transferred to the Redis-1s slave, and the status is +sdown.

[28156] 00:42:48.431 # +sdown masterserver-1m 192.168.84.129 6379

[28156] 00:42:48.431 # +odown masterserver-1m 192.168.84.129 6379 #quorum 1/1

[28156] 00:42:56.570 # +failover-detected masterserver-1m 192.168.84.129 6379

[28156] 00:42:56.670 # +failover-end masterserver-1m 192.168.84.129 6379

[28156] 00:42:56.670 # +switch-masterserver-1m 192.168.84.129 6379 192.168.84.128 6389

[28156] 00:42:57.055 * +sentinel Sentinel 192.168.84.128:26389 192.168.84.128 26389 @server -1m 192.168.84.128 6389

[28156] 00:43:01.688 # +sdown slave 192.168.84.129:6379 192.168.84.129 6379 @server -1m 192.168.84.128 6389

Iv. Backup Recovery

1. Backup strategy

Redis offers two relatively effective backup methods: Rdb and AOF.

An RDB is a snapshot of all the data in memory that is saved to disk at a point in time, and when data is recovered, all data before the backup time can be restored, but the data behind the backup points cannot be recovered.

AOF is the purpose of logging database state by logging all commands (and their parameters) that have been written to the database in the form of protocol text to the AOF file. The advantage is that you can basically achieve no loss of data (cached data may be lost), the disadvantage is that as the volume of data continues to increase, aof files will become larger.

In the case of data security, as far as possible to avoid the backup data consumption too much Redis resources, the following backup strategy:

Master side: No backup mechanism is used

Slave: Use AOF (the RDB can be turned on when strict data is required) to back up aof files to the backup server daily.

To minimize resource disruption at the master end, migrate all of the backups to the slave end. At the same time there are shortcomings, when master hangs, the application service to switch to the slave side, at this time the slave side of the load will be very large. Currently Redis does not support RDB and aof parameter dynamic modification, need to restart Redis to take effect, want to be able to achieve more efficient modification in the new version.

2. Disaster recovery

          When the master-side Redis service crashes (including host power outages, process disappearance, and so on), Redis Sentinel switches slave to read-write status and provides production services. With troubleshooting, Master is automatically added to Sentinel after startup and data synchronization is done from the slave side, but does not switch.

· when master and slave crash at the same time (such as the engine room power down), after starting the server, the backup server the latest AOF backup copy to the master side, start master. After everything is done, start slave.

------------------------------------------------------------------

To view Redis information:

Redis-cli-h 127.0.0.1-p 6379 Info Replication

# Replication
Role:master
Connected_slaves:1
Slave0:10.0.2.212,7379,online

To view Sentinel information:

Redis-cli-h 172.18.18.207-p 26379 info (followed by # bytes, showing only the contents of the specified section)

Info "command will print the full service information, including the cluster, we only need to focus on the" Replication "section, which will tell us the" role of the current server "and all the slave information that points to it. Can be used on any one of the slave, using the" Info "command to get the master information pointed to by the current slave

You can use REDIS-CLI to view Sentinel-managed Redis clusters:

Redis-cli-h 172.18.18.207-p 26379 Sentinel Masters

1) "Name"

2) "MyMaster"

3) "IP"

4) "172.18.18.207"

5) "Port"

6) "6500"

...

View a specified master with those slaves:

172.18.18.207:26379> Sentinel Slaves MyMaster

1) "Name"

2) "172.18.18.207:6501"

3) "IP"

4) "172.18.18.207"

5) "Port"

6) "6501"

...

You can also force a Redis cluster to do failover:
172.18.18.207:26379> Sentinel failover MyMaster

Ok

Sentinel-based Redis high-availability scenario