Applications. Net + Consul maintain the high availability of RabbitMq, consulrabbitmq

Applications. Net + Consul maintain the high availability of RabbitMq, consulrabbitmq

　　The process of studying lazy people is to let the boss do what they want to do and study what they want. On weekends when the National Day holiday is back, the boss assigned me a task through dingtalk, the high availability solution of RabbitMQ, I want to say that dingtalk is too pitfall:

This is the task that I got off work on Sunday evening on the 9th day after the National Day. I saw it on Monday. Next Friday, isn't it the 21st? The time is so abundant! Isn't it too early .. I happened to have an interview for dinner on the evening of 9th. Then I asked me a few algorithms and I was despised .. He said that a front-end is better than a back-end algorithm. Please treat me -. -So the optical algorithm went from Monday to Wednesday (Where are some cooperations for programmers to brag about?) until Thursday, the boss said that the task will expire tomorrow! What's wrong with the study! Now, I suddenly realized that dingtalk is a pitfall for you. Next Friday is the 14th !!

RabbitMQ high-availability cluster Simple configuration

　　I think this article is quite detailed about the high-availability cluster of RabbitMQ. To adopt the so-called Active/Active solution, you can only select the image mode (Only supported by version 3.x or later). Another explanation is provided. (in contrast to a common cluster, the essence is that the message entity actively synchronizes data between mirror nodes, rather than pulling data temporarily when the consumer retrieves data. This mode also has obvious side effects. In addition to reducing system performance, if there are too many mirror queues and a large number of messages enter, the network bandwidth inside the cluster will be greatly consumed by such synchronous communication. So it is applicable in scenarios with high reliability requirements). After setting up the RabbitMq cluster,

In image mode, you can add a synchronization policy through the command action queue, for example

Apply mirroring policies to all queues

Rabbitmqctl set_policy ha-all "^" '{"ha-mode": "all "}'

Or specify the queue Name:

Rabbitmqctl set_policy yu-ha "^ yu" '{"ha-mode": "exactly", "ha-params": 2, "ha-sync-mode ": "automatic "}'

Or directly add

Click the Admin menu --> the Policies option on the right --> Add/update a policy at the bottom left

 

 

Name is the queue name, Pattern is the matching rule, for example, to write a ^ yu is the queue starting with yu, ha-mode = What queue will be synchronized, for example, if it is set to all, all matching queues are synchronized.

Then, when creating a new queue, you can specify the host column on the machine.

So-called pitfalls

To copy the content of this article, the common method is to ensure high availability of RabbitMq clusters through HAProxy + KeepAlive:

The process of creating a queue:

Assume that node2 is down:

• Node 2 no longer responds to the heartbeat and is considered to have been removed from the cluster.
• Master queue on node 2 is no longer available
• RabbitMQ upgrades the salve instance on node 1 or 3 to the master instance.

Assume that the master queue is still on node 2, and the client accesses the queue through LB:

It can be seen that in this configuration, 2/3 of client requests need to be redirected, which may cause a high probability of access delay, but the access will still succeed. There are two methods to optimize:

• Directly connect to the node where the master queue is located, so there is no need to redirect. However, for this method, you need to calculate in advance and then tell the client which node has the master queue.
• Deploy queues evenly among all nodes to reduce the probability of redirection.

To avoid such n-1/n redirection, it is very important to know the node where the Master queue is located.

Ideas

This figure roughly means:

 

　　1.Register RabbitMq to Consul (step 1) and perform health monitoring on RabbitMq through Consul. Meanwhile, Consul provides the configuration center service to store configuration information about RabbitMq, such as the queue account, password, queue host name, Ip address, etc. For example:

Register the RabbitMq service to Consul:

{   "services": [{     "id":"rabbit@rabbitmq1",     "name":"RabbitMqServer",     "tags":["rabbitMq"],     "address": "192.168.1.101",     "port": 15672,     "checks": [       {         "Http": "http://192.168.1.101:15672/",         "interval": "10s"       }     ]   },   {     "id":"rabbit@rabbitmq2",     "name":"RabbitMqServer",     "tags":["rabbitMq"],     "address": "192.168.1.102",     "port": 15672,     "checks": [       {         "Http": "http://192.168.1.102:15672/",         "interval": "10s"       }     ]   }   ] } 

View Code

Store the queue information of RabbitMq to Consul:

　　2.For business services, you must configure QueueName + VirthHost and obtain queue information from the RabbitMq gateway through step 2. Then, you can perform the push-pull Operation with the queue through Step 5, you can obtain the node information of the Master queue corresponding to the available queue to avoid the issue of receiving push and forwarding such as n-1/n.

　　3.After the RabbitMq gateway receives a request from the service, it obtains the information of any healthy RabbitMq queue in the cluster through Consul (health monitoring function provided by Consul ),Then obtain the WebApi that communicates with RabbitMq Based on the queue. The Http Api documentation of RabbitMq provides an interface for obtaining queue details.For example, the Master information used to obtain the queue pair is available through the interface: http: // 127.0.0.1: 15672/api/queues/% 2F/yu_queue, % 2F corresponds to VirthHost, yes/transcoding. yu_queue is the queue name. These two parameters are provided when the service requests the Api. The Json string returned by the Api contains the peer information of the Master node of the queue, the node attribute corresponds to the node name of rabbitmq, for example, rabbit @ rabbitmq1, then, you can use the RabbitMq information configured in consul to find the queue information used by the node and return it to the service.

Note the following:

Identity authentication is required to access the Http Api of RabbitMq. the Token obtained for Basic authentication is not found in the query documentation --

　　

Then I was surprised to find out ..Convert. ToBase64String (Encoding. ASCII. GetBytes (userName + ":" + password )));Ah .. Don't want to say more...

There is another issue with/in Api parameters. There is no problem with HttpClient for. net 4.5 or later versions. When using HTTP webrequest or later versions, you need to process the Uri.

public static void ForceCanonicalPathAndQuery(Uri uri)        {            string paq = uri.PathAndQuery; // need to access PathAndQuery             FieldInfo flagsFieldInfo = typeof(Uri).GetField("m_Flags", BindingFlags.Instance | BindingFlags.NonPublic);            ulong flags = (ulong)flagsFieldInfo.GetValue(uri);            flags &= ~((ulong)0x30); // Flags.PathNotCanonical|Flags.QueryNotCanonical             flagsFieldInfo.SetValue(uri, flags);        } 

Additional description of View Code:

In fact, the idea is very simple, but it is clear that you can encapsulate an SDK (and you have to package--) to complete things. Why do you need to involve Consul and a redundant RabbitMq Gateway, what's more, with the RabbitMq gateway, isn't it a single point of failure ?!

　　For single point of failure.. I think there is no pressure to deploy several stateless gateways .. Go to training in the SKD package!

　　Why is Consul used?On the one hand, for health monitoring, health monitoring allows Consul to directly obtain available Http Api Information of RabbitMq through the Consul Http Api, as well as the configuration center. The service configures the queue name, VirthHost, and get the RabbitMq gateway of the queue service. The RabbirMq gateway obtains the queue information through the configuration center. The data in the configuration center is dynamic and easy to update.

　　Why do I use the gateway as an intermediate proxy instead of directly connecting to RabbitMq through the SDK?When obtaining queue information through Consul, you can make a regular cache, and the information such as the queue's username and password is not maintained through the service configuration. if the service is obtained through the SDK directly through Consul, dependencies also become slightly more complex. The RabbitMq Api can also dynamically obtain the cluster node information and permission information of the queue. Regular updates in the service SDK are not a problem, but the SDK becomes more complex after all, are you tired of doing that ..

　　Why does the parameter need VirthHost + QueueName?Different departments are not the same person ..

Tangled issues:

To address this problem, the Routing mode and Topic mode in RabbitMq may be faulty. When a single routekey corresponds to a single queue, the routekey In Exhange can be obtained through the queue to be effectively sent to the queue on a server, if the queue bound to the routekey is distributed across multiple servers and the master and slave queues are distributed across multiple servers, the obtained "Master queue" can only be used for a real Master instance in a queue. If the Master node is not in the ip address of other queues, the forwarding problem still exists. There is no problem in pulling data (I need the queue name to pull data !), When pushing data, only Exchange + RouteKey is used. Who cares if your queue is master or slave?

 

The code is being organized ....

 

Reference link:

Http://www.cnblogs.com/sammyliu/p/4730517.html

Https://insidethecpu.com/2014/11/17/load-balancing-a-rabbitmq-cluster/