Selection practice of main preparation scheme of message middleware

Background Introduction

The message middleware consists of three subsystems: the Client SDK, the server, and the Control Management Center (admin). What we call the main preparation is mainly for the server, and the server node is locally stateful, in particular, publish stateless, but pull stateful. Therefore, for a release, because of the distributed deployment itself is highly available, but for the pull, there is a single point of availability issues. key issues in primary and standby switching

There are two core issues that need to be addressed to achieve a master and standby switchover:
1. Resolve State detection, communication, and decision-making issues between primary and standby.
Through heartbeat detection to know each other's state, once the other side of the state abnormal need to take over the other side of the resources to continue to provide services.
2. Resolve the issue of resource transfers.
The problem of resource transfer can be solved by resource sharing mechanism or resource synchronization redundancy method.

Any set of primary and standby switching solutions will need to address these two basic issues. Comparison of fault detection status and decision-making resource switching schemes for Main and standby switching Zookeeper

If you use zookeeper, each group of primary servers needs to deploy zookeeper as zookeeper Server;sdk as the zookeeper Client, establish a connection to the primary and standby server, and monitor zookeeper The server event. If this occurs, the primary and standby switch can be triggered by an event that is heard. In fact, we only solved the first problem using zookeeper. As for the second question, if it is a single mode, the resource transfer can be solved by using the DB middleware to share the resource, and if it is Master-slave mode, it needs to be developed by the server itself. load Balancer Software (LVS, HAProxy, Nginx) + Highly available software (keepalived, Heartbeat)
This scenario is high availability for load balancing server, keepalived and heartbeat only solve the first problem, and the second problem is load balancer software to solve.

Self-Research
Self-research programme I Admin as an observer, establish a connection to each of the primary and standby servers and monitor the connection status through heartbeat detection. Admin manages each set of primary and standby relationships and their activation status. You need to consider the high availability of the admin itself. The server's standby startup item differs from the host and only provides the pull service, and other services are denied. In addition, the standby machine in the recovery, you need to stop the service, and then synchronize the consumption status; When the server's host recovers, it first pulls the consumption status information from the admin and updates it to DB. The SDK periodically pulls the information of the pigeon server from the Fly Pigeon admin, if found slave is activated, establishes the connection with slave, rejects the master connection, and if the master is found to be active, establishes a connection to master and, if the connection is established successfully, Remove the slave connection again.

Master and Standby switch sequence diagram:

Master and Standby switch sequence diagram:

Self-Research program two add a pigeon proxy between the SDK and the server, which is responsible for the primary and standby switching, and is responsible for routing, so that the SDK can be lightweight. Because the server connection is currently authenticated, it is necessary to move the authentication function to the proxy. Or consider whether the state of the connection can be removed. Whether the connection has a state will affect the proxy itself high-availability scheme, if the connection is stateless, the direct use of F5, if there is a state, the proxy itself is a high-availability solution is a problem. Proxy to maintain this and master and slave long connection, and do heartbeat detection, detection to achieve the switching conditions, then trigger the switching action. The proxy scheme has a great impact on the existing flying Pigeon scheme, and the change surface is more.