Application. Net+consul maintaining high availability of RABBITMQ

　　Lazy learning process is the work of the eldest brother let do what to do research and research what, National Day holiday back weekend boss through nail to me decorate a task, RABBITMQ High available solution , I want to say nail too pit:

This is the national day after the 9th Sunday night commute to the task, I saw in Monday when a look, next Friday, it is not 21st, time is so abundant! It's not too early. It happened that the students to interview 9th night with dinner, and then asked me a few algorithms, and then despised. He says I have a front-end algorithm that's better than you, you treat it. -So Monday to Wednesday optical algorithm (programmer in order to blow, which have what moral integrity AH) until Thursday boss said, tomorrow task expires! How's the research? At this time only suddenly dawned, nail you a pit cargo, next Friday is 14th number!!

RabbitMQ High-availability clusters Simple Configuration

　　For the description of RABBITMQ high availability cluster, I think this article is very detailed, do not say. Configure the way the cluster crossing network can be, in order to take the so-called active/active scheme, so we can only select the Mirror mode (3.x version above is supported ). Another explanation. The difference between the substance and the normal pattern is that, compared to the normal cluster, The message entity is actively synchronizing between mirror nodes, rather than being temporarily pulled when the consumer is fetching data. The side effects of this pattern are also obvious, in addition to reducing system performance, if the number of mirror queue, coupled with a large number of messages, the network bandwidth within the cluster will be greatly consumed by this synchronous communication. Therefore, in the case of high reliability requirements, when the RABBITMQ cluster is built,

Mirroring mode can add synchronization policies through the command behavior queue, such as

Policies that apply mirroring mode to all queues

Rabbitmqctl set_policy ha-all "^" ' {"Ha-mode": "All"} '

or specify the name of the specified queue:

Rabbitmqctl set_policy yu-ha "^yu" ' {"Ha-mode": "Exactly", "Ha-params": 2, "Ha-sync-mode": "Automatic"} '

Or add them directly to the admin page

Click the Admin menu --The policies option on the right--the bottom left add/update a policy

Name is the name of the queue, pattern is a matching rule, such as write a ^yu is Yu start queue, ha-mode= What is to synchronize what queue, such as =all words is to synchronize all matching queue.

Then you can designate that machine to run on the home row when you create a new queue.

the so-called pit

or copy the content of this article, the common means is to ensure that the RABBITMQ cluster is highly available through Haproxy+keepalive:

The process of creating a queue:

1. LB distributes the client request to node 2,client to create the queue "Newqueue" and begins to put a message into it.
2. Eventually, the backend service creates a snapshot of "Newqueue" on Node 2 and copies it to Node 1 and 3 over a period of time. At this point, the queue on the node2 is the slave queue on the master Queue,node 1 and 3 queues.

If now Node2 down:

• Node 2 no longer responds to the heartbeat, it is thought to have been removed from the cluster
• The Master queue on Node 2 is no longer available
• RabbitMQ Upgrade the Salve instance on node 1 or 3 to master instance

Assuming that the master queue is still on Node 2, the client accesses the queue through LB:

1. The client connects to the cluster to access the "Newqueue" queue
2. LB distributes requests to a single node based on the configured polling algorithm
3. Suppose the client request is transferred to Node 3
4. RabbitMQ found "Newqueue" Master node is node 2
5. RabbitMQ the message to Node 2
6. The end client successfully connects to the master queue on Node 2

It can be seen that in this configuration, 2/3 of client requests require redirection, which results in a large probability of access delays, but ultimately the access will be successful. To optimize, there are two ways:

• Connect directly to the node where the master queue resides so that no redirection is required. But in this way, it needs to be calculated in advance and then tell the client which node has the master queue.
• Distribute queues evenly across all nodes as much as possible, reducing the probability of redirection

To avoid this redirection of n-1/n, it is important to know the node where the master queue resides, and then not copy it.

Ideas

The general meaning is this picture:

　　1. Register RABBITMQ in Consul (step 1), RABBITMQ health monitoring through Consul, and consul provide configuration center services, can store some RABBITMQ configuration information, such as queue account, password, Queue hostname, location IP, etc., for example:

To register the RABBITMQ service with consul:

{   "Services": [{     "ID":"[email protected]",     "name":"Rabbitmqserver",     "Tags":["RabbitMq"],     "Address":"192.168.1.101",     "Port":15672,     "Checks": [       {         "Http":"http://192.168.1.101:15672/",         "interval":"10s"       }     ]   },   {     "ID":"[email protected]",     "name":"Rabbitmqserver",     "Tags":["RabbitMq"],     "Address":"192.168.1.102",     "Port":15672,     "Checks": [       {         "Http":"http://192.168.1.102:15672/",         "interval":"301"       }     ]   }   ] } 

View Code

To deposit the RABBITMQ queue information into the consul:

　　2. The business service needs to be configured with Queuename+virthhost, to obtain the queue information from the RABBITMQ gateway through step 2, and then to push and pull through step 5 with the queue. Here you can get the node information for the master queue corresponding to the available queue, and avoid n-1/n this issue of accepting push forwarding.

　　3.after the Rabbitmq Gateway accepts a request for a business service, consul obtains information about any healthy RABBITMQ queue in the cluster (consul provides health monitoring functions), Then, based on the queue, the HTTP API documentation for the WEBAPI,RABBITMQ that communicates with the RABBITMQ provides an interface to get the queue details , such as obtaining the master information for the queue to be available via interface:/HTTP 127.0.0.1:15672/api/queues/%2f/yu_queue,%2f corresponds to Virthhost, which is the transcoding, Yu_queue is the queue name, these two parameters are provided through the Business Service Request API, The JSON string returned through the API contains the alignment information for the node of the queue master, where the node attribute corresponds to the RABBITMQ name, such as: [email protected], You can then use the RABBITMQ information that you previously configured in Consul to find the queue information that the node is using to return to the business service.

There's going to be a hole here. Note:

Access to the HTTP API for RABBITMQ is required for authentication, and this Basic authentication token gets checked for documentation not found--

　　

Later surprised to discover that. convert.tobase64string (Encoding.ASCII.GetBytes (UserName + ":" + password )); Hey.. Don't want to say more ...

There is a problem with the API parameters, the. NET 4.5 version with HttpClient no problem, 4.5 The following version or the use of HttpWebRequest when the URI should be processed

 Public Static voidforcecanonicalpathandquery (Uri uri) {stringPaq = URI. Pathandquery;//need to access PathandqueryFieldInfo Flagsfieldinfo =typeof(Uri). GetField ("M_flags", BindingFlags.Instance |bindingflags.nonpublic); ULONGFlags = (ULONG) Flagsfieldinfo.getvalue (URI); Flags&= ~ ((ULONG)0x30);//flags.pathnotcanonical| Flags.querynotcanonicalflagsfieldinfo.setvalue (URI, flags); } 

View CodeAdditional Instructions:

In fact, the idea is very simple, but clearly can be encapsulated by a SDK (and it would have to be encapsulated--) on the completion of the matter why also involved in the consul and redundant a RABBITMQ gateway, and there is RABBITMQ gateway is not to say a single point of the problem?!

　　for a single point of issue : I think there is no pressure to deploy a few stateless gateways. SKD package when you rotation go!

　　for why use Consul, on the one hand is for health monitoring, health monitoring can let consul through consul HTTP API directly get the available RABBITMQ HTTP API information, there is the configuration center, business services by configuring the queue name, Virthhost, and get the queue service RABBITMQ Gateway, RABBIRMQ Gateway is responsible for the configuration center to obtain the queue information, Configuration center data is dynamic, update is more convenient.

　　for why the API is not directly connected through the SDK Rabbitmq, but through the gateway as an intermediary agent , the queue information through consul can be a timed cache, and such as the queue of user name password such information through the business service configuration is not easy to maintain, Business services are acquired through the SDK directly through consul, and dependencies can become slightly more complex. Through the RABBITMQ API can also dynamically get the queue of cluster node information, permissions information, etc., in the Business Services SDK regularly updated is also possible, but the SDK has become complex, you do that much not tired?

　　why the parameters to Virthhost+queuename, different departments of people are not the same person.

The problem being tangled:

For this idea, for RABBITMQ in the routing mode and topic mode will be problematic, single routekey corresponding to a single queue can be obtained through the queue exhange the Routekey is effectively sent to a queue on a server, If the Routekey-bound queue is distributed across multiple servers and the master-slave distribution of these queues is on multiple servers, I can get to the "master queue" only for a certain queue when the true master, for other queues if Master is not on the IP or there is a forwarding problem. This problem does not matter when pulling the data (pull the data I need the queue name!) ), push data only with Exchange+routekey, who still cares if your queue is the master from?

The code is tidying up ....

Reference Links:

Http://www.cnblogs.com/sammyliu/p/4730517.html

https://insidethecpu.com/2014/11/17/load-balancing-a-rabbitmq-cluster/

Application. Net+consul maintaining high availability of RABBITMQ