Question proposal
Redis (specifically 2.8.14 and below) replication only supports master-slave replication. In the actual production environment, this one-way master-slave replication cannot be highly available (of course, if data loss is allowed, you can adopt keepalived and the policy_master/policy_slave mechanism, forced master-slave role swaps. In this way, unsynchronized data will be completely lost in the process of forced master-slave swaps, which is an extremely dangerous solution, ).
The so-called stone in other mountains can attack the jade. Mysql provides mature master-master replication. Combined with keepalived dynamic IP addresses, it can provide services at the same time at two nodes (ready). When either node fails, the other node immediately takes over seamlessly. After the server is started, the data that has not been synchronized will continue to be synchronized, ensuring that the data is not lost to the maximum extent. Of course, this solution is not 100% consistent, because after the nodes are hung up, there may be a sequential problem of Operation columns, resulting in a small amount of data inconsistency. The more data is not synchronized, the possibility of inconsistency is higher, but in production applications, this solution can ensure high availability to the maximum extent, and the impact on consistency is not particularly serious, so this solution is widely used.
Therefore, to achieve high availability of redis, you must first solve the master-master replication problem. Gleasy initially implemented a proxy-based redis cluster solution to solve this problem. For details, refer to this article "Gleasy NOSQL database cluster Cloudredis". This cluster solution has been working in the production environment for a period of time and exposes a problem. The nodes in the cluster cannot achieve full data consistency, inconsistencies exist in those expired keys (with expire set and expired ). Redis's key failure mechanism is divided into active failure and passive failure, and active failure only takes a small part of Random failure each time. When the number of expired keys is large, these expired keys will always exist, and all nodes in the cluster will lose part of them at random, resulting in different keys that are invalid for these nodes. The final result is data inconsistency, this inconsistency will cause great confusion to the O & M work, because I don't know whether the synchronization mechanism is abnormal or the KEY is invalid, so that O & M personnel are always restless. Finally, we decided to provide another mechanism to implement the real redis master service.
Implementation principle
1. Overview
Install a module (we call it a Cluster module) on each redis node, which can obtain all the write operation command sequences of the redis node;
The Cluster module obtains all the write operation sequences of redis nodes and writes them to the binlog file;
From the Cluster module of the node, request the binlog block from the Cluster module of the master node, and record the location of the last request. The next request is followed by the last location;
Obtain the binglog block from the Cluster module of the node, analyze the write operation sequence command, and execute it from the redis node.
2. Specific implementation
The Cluster module exists as an independent application and is named rediscluster (The first reason for making an independent process is to ensure better performance and avoid adding it to the redis single-thread event mechanism, second, avoid the inconvenience caused by too many changes to the redis source code );
Rediscluster listens to independent ports, and rediscluster communicates with each other through independent ports;
Each redis node corresponds to a rediscluster. After the rediscluster is started, it acts as the redis slave and receives all the write operation command sequences in real time;
Rediscluster starts an independent thread and regularly accesses the port of the primary node rediscluster to obtain binlog data;
3. Some key technical points
3.1 consistency assurance and technical implementation
Use the following constraints to ensure consistency:
A. When the rediscluster is not ready (that is, the operation command is not correctly received and written to the binlog), redis cannot accept the write operation. That is, the write operation service is not provided if the binlog write operation is unsuccessful.
Implementation: set the parameter min-slaves-to-write of redis to 1, which ensures that write is allowed only when at least one slave works well.
B. The active invalidation function of expired Keys is not enabled for redis on the Slave node. Only the primary node enables the active invalidation function of expired keys.
Implementation: When the rediscluster receives the binlog, the active invalidation function of redis is disabled; when the rediscluster receives the write command from redis, the active invalidation function of redis is enabled;
C. The Master node and Slave node cannot be written at the same time (one write, and the other automatically becomes read-only)
Implementation: redis write operations are blocked when rediscluster receives binlog;
D. The Write command of the master node received by slave will not be sent to rediscluter again.
Implementation: rediscluster sends the binlog off command to redis, and then receives the binlog and writes it to redis. redis will not send it to slave.
E. Add the unique identifier of the server to all commands to avoid endless loops.
Implementation: The unique identifier of the server is added when rediscluster is written to the binlog file. When a master-slave ring is formed, ignore the self-generated binlog based on the unique identifier and only consume the binlog generated by others.
3.3 Transformation of redis
A. Enable/disable the active failure function
Introduce the backup on/off command to disable/enable this function.
B. Write blocking
Introduce the command lock on/off to block/unblock write operations
C. Enable/disable the binlog function of the connection
Binlog on/off is introduced to enable/disable the binlog function of the connection (after the function is disabled, all write operations will not be sent to slave)
