A step by step realization of hot standby of Redis+sentinel double machine

Preface

A few days have been busy online environment deployment of things, the initial thought is that Nginx (KeepAlive dual standby) +3 (Tomcat) +2redis (dual-standby), but later because the Aliyun Server Classic network does not provide virtual IP, can not use keepalive, Nginx Dual Machine Hot standby can only temporarily give up, the second step, the use of Nginx+3tomcat+2redis (dual-standby). Nginx+tomcat because of the previous configuration, so the focus on the Redis dual-machine hot standby, after all, is the online system, the appropriate resilience or need, I can not like the test system so go to play, otherwise the blame on some back, after all, code codes to earn some living expenses is not easy.

Also looked up some data on the net, Redis cluster implementation probably has the following several ways:

1.redis-cluster, the official cluster-building program (too heavyweight, more suitable for the use of the time when the larger data volume)

2.redis+keepalive (because we use the Aliyun server does not support virtual IP, so this package is aborted)

3.redis+zookeeper (need to introduce zookeeper, large changes to existing code)

4.redis+sentinel (Redis self-monitoring middleware) (small code changes, less configuration, and can meet the needs of dual-machine hot standby)

Based on our current situation and requirements, after a brief comparison, our team decided to choose a fourth option Redis+sentinel to achieve dual-machine hot standby. preparatory work 1. Install redis-001 (main)

$ wget http://download.redis.io/releases/redis-3.2.3.tar.gz
$ tar xzf redis-3.2.3.tar.gz
$ mv redis-3.2.3 redis-001
$ cd redis-001
$ make
2. Install redis-002 (from)
$ wget http://download.redis.io/releases/redis-3.2.3.tar.gz
$ tar xzf redis-3.2.3.tar.gz
$ mv redis-3.2.3 redis-002
$ cd redis-002
$ make
3. Modify the redis-002 port to 6380
$ cd / usr / tools / redis-002 /
$ vim redis.conf
... find port 6379 to 6380 ...
4. Modify the redis default configuration
a. Close the IP binding, and note the bind 127.0.0.1 in redis.conf

b. Turn off protected mode and change protected-mode yes to protected-mode no 5. Start redis-001, redis-002

$ cd tools / redis-001 /
$ ./src/redis-server redis.conf
The following information indicates that the startup was successful:

Step by step implementation of redis + sentinel hot standby

12513: M 09 Oct 11: 15: 49.061 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory = 1' for this to take effect.
12513: M 09 Oct 11: 15: 49.061 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never> / sys / kernel / mm / transparent_hugepage / enabled 'as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
12513: M 09 Oct 11: 15: 49.061 * DB loaded from disk: 0.000 seconds
12513: M 09 Oct 11: 15: 49.061 * The server is now ready to accept connections on port 6379
Redis master-slave replication 1.Redis replication characteristics
    1). The same master can synchronize multiple slaves.
    2). Slave can also accept connection and synchronization requests from other slaves, which can effectively offload the synchronization pressure of the master. So we can think of Redis's Replication architecture as a graph structure.
    3). Master Server provides services for Slaves in a non-blocking manner. So during Master-Slave synchronization, the client can still submit queries or modify requests.
    4). Slave Server also completes data synchronization in a non-blocking manner. During synchronization, if a client submits a query request, Redis returns the data before synchronization.
    5). In order to offload the read operation pressure of the master, the slave server can provide read-only operation services for the client, and the write service must still be completed by the master. Even so, the scalability of the system has been greatly improved.
    6). The Master can hand over the data saving operation to Slaves, thereby avoiding the need for an independent process in the Master to complete this operation. 2.Redis replication principle

    After the slave starts and connects to the master, it will actively send a SYNC command. After that, the master will start the background save process and collect all the received commands for modifying the data set. After the background process is completed, the master will send the entire database file to the slave to complete a complete synchronization. After receiving the database file data, the Slave server saves it to disk and loads it into memory. After that, the master continues to send all the modified commands that have been collected, and the new modified commands to Slaves in turn, and Slave will execute these data modification commands this time to achieve the final data synchronization.
    If the link between the master and the slave is disconnected, the slave can automatically reconnect to the master, but after the connection is successful, a full synchronization will be performed automatically. 3.redis master-slave configuration

As mentioned earlier, the master-slave configuration of redis is very simple, and only one slaveof is required.

Connect to redis-002 and execute slaveof 192.168.231.130 6379:

C: \ Users \ lenovo \ Desktop \ redis> D:
D: \> cd D: \ Program Files \ Redis-x64-3.0.5
D: \ Program Files \ Redis-x64-3.0.5> redis-cli.exe -h 192.168.231.130 -p 6380
192.168.231.130:6380> slaveof 192.168.231.130 6379
OK
192.168.231.130:6380>
After the execution is complete, you can see the redis-001 console output on the server:

12513: M 09 Oct 11: 15: 49.061 * DB loaded from disk: 0.000 seconds
12513: M 09 Oct 11: 15: 49.061 * The server is now ready to accept connections on port 6379
12513: M 09 Oct 11: 41: 59.371 * Slave 192.168.231.130:6380 asks for synchronization
12513: M 09 Oct 11: 41: 59.371 * Full resync requested by slave 192.168.231.130:6380
12513: M 09 Oct 11: 41: 59.371 * Starting BGSAVE for SYNC with target: disk
12513: M 09 Oct 11: 41: 59.372 * Background saving started by pid 13673
13673: C 09 Oct 11: 41: 59.388 * DB saved on disk
13673: C 09 Oct 11: 41: 59.388 * RDB: 6 MB of memory used by copy-on-write
12513: M 09 Oct 11: 41: 59.462 * Background saving terminated with success
12513: M 09 Oct 11: 41: 59.462 * Synchronization with slave 192.168.231.130:6380 succeeded
redis-002 console output:

13581: S 09 Oct 11: 41: 59.361 * Connecting to MASTER 192.168.231.130:6379
13581: S 09 Oct 11: 41: 59.370 * MASTER <-> SLAVE sync started
13581: S 09 Oct 11: 41: 59.370 * Non blocking connect for SYNC fired the event.
13581: S 09 Oct 11: 41: 59.371 * Master replied to PING, replication can continue ...
13581: S 09 Oct 11: 41: 59.371 * Partial resynchronization not possible (no cached master)
13581: S 09 Oct 11: 41: 59.374 * Full resync from master: e2f4e2608956ea6392482c2e0a9429efdebd2b53: 1
13581: S 09 Oct 11: 41: 59.462 * MASTER <-> SLAVE sync: receiving 76 bytes from master
13581: S 09 Oct 11: 41: 59.463 * MASTER <-> SLAVE sync: Flushing old data
13581: S 09 Oct 11: 41: 59.463 * MASTER <-> SLAVE sync: Loading DB in memory
13581: S 09 Oct 11: 41: 59.463 * MASTER <-> SLAVE sync: Finished with success
But if you do this, it will be restored after restarting redis-002, so usually we will directly modify the redis-002 configuration file redis.conf, add slaveof 192.168.231.130 6379 at the end, and it will be permanent.

Now we write data to redis-001, and we can query it from redis-002. Let's test it, and write data with key name and value osc into redis-001.

192.168.231.130:6379> set name osc
OK
192.168.231.130:6379> keys *
1) "name"
192.168.231.130:6379>
Querying the data in redis-002, you will find that the key of name also exists, and the value is osc

192.168.231.130:6380> keys *
1) "name"
192.168.231.130:6380> get name
"osc"
192.168.231.130:6380>
So far, the redis master-slave replication has been successfully configured. The next step is to start the key work and configure dual-system hot backup. Redis + sentinel hot standby

    Dual-machine hot standby refers to hot standby (or high availability) based on two servers in a high-availability system. Because two-machine high availability is used more in China, it is named dual-machine hot standby. The switching methods are divided into active-standby mode (Active-Standby mode) and dual-master mode (Active-Active mode). Active-standby mode refers to a server in an active state (that is, Active state) , The other server is in the standby state of the service (that is, the Standby state). The dual-host mode refers to the active and standby states of two different services on two servers, respectively (that is, Active-Standby and Standby-Active states).

The vernacular is that when the master server hangs up, the slave server immediately switches to the master server to continue working. When the original master server is fully repaired and started, it will automatically act as the slave server to continue working. This is a good way to avoid the phenomenon of system hanging points due to a host failure.

Sentinel (Sentinel) is a tool for monitoring the status of the master in the redis cluster, which has been integrated into the official redis version Can be directly configured and used. 2.Sentinel command

       PING: Returns PONG.
       SENTINEL masters: List all monitored master servers and their current status;
       SENTINEL slaves <master name>: list all slave servers for a given master server, and the current status of these slave servers;
       SENTINEL get-master-addr-by-name <master name>: Returns the IP address and port number of the master server with the given name. If this master server is performing a failover operation, or the failover operation for this master server has been completed, then this command returns the IP address and port number of the new master server;
       SENTINEL reset <pattern>: Reset all master servers whose names match the given pattern. The pattern parameter is a Glob-style pattern. The reset operation clarifies all the current status of the master server, including the ongoing failover, and removes all slave servers and Sentinel of the master server that have been found and associated so far;
       SENTINEL failover <master name>: When the master server fails, forcibly start an automatic failover without asking other Sentinel opinions.

The client can obtain the current master server IP address and port number through SENTINEL get-master-addr-by-name <master name> and SENTINEL slaves <master name> to obtain all slaves information

a. Close redis-001 and redis-002

b. Add the following configuration information to redis-001 and redis-002 configuration sentinel.conf:

sentinel monitor mymaster 192.168.231.130 6379 1
c. Modify the redis-002 sentinel.conf listening port:

$ Cd / usr / tools / redis-002 /
$ vim sentinel.conf
... find port 26379 to 26380 ...
port 26380
d. Start redis-001 (master) and redis-002 (slave), redis service and sentinel service respectively

$ Cd / usr / tools / redis-002 /
$ ./src/redis-server redis.conf
$ ./src/redis-sentinel sentinel.conf
$ Cd / usr / tools / redis-001 /
$ ./src/redis-server redis.conf
$ ./src/redis-sentinel sentinel.conf
At this time, in the master-slave redis, the following message appears in the sentinel console:

14677: X 09 Oct 14: 04: 49.633 # Sentinel ID is 7b8fdc1e5e47426b0d62a3ddd22ede0fd712f452
14677: X 09 Oct 14: 04: 49.633 # + monitor master mymaster 192.168.231.130 6379 quorum 1
14677: X 09 Oct 14: 04: 49.634 * + slave slave 192.168.231.130:6380 192.168.231.130 6380 @ mymaster 192.168.231.130 6379
14677: X 09 Oct 14: 05: 53.727 * + sentinel sentinel 31e660153984b606951c564395bdc8e193943f0b 192.168.231.130 26380 @ mymaster 192.168.231.130 6379
e. Test results:

Connect to the master server sentinel. Note that the port is the port configured in sentinel.conf.

Query the master server information according to the sentinel command described earlier:

192.168.231.130:26379> SENTINEL masters
1) 1) "name"
    2) "mymaster"
    3) "ip"
    4) "192.168.231.130"
    5) "port"
    6) "6379"
    7) "runid"
    8) "fafb94fe9bff119e8a97f9183e3bcb5561933631"
    9) "flags"
   10) "master"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"