RabbitMQ High Availability cluster construction and e-commerce platform use experience summary

1. Design your message in an EDA (event-driven architecture)-oriented way

2. The design of AMQP routing key

3. RabbitMQ Cluster Construction

4. Mirror Queue policy settings

5. Two good RABBITMQ plugin large application plugins (sharding, rederation)

6. Queue mirroring failed manual synchronization

7. Synchronization mode for each cluster configuration (RabbitMQ export\import)

8. Client connection mode (use AMQP groups as much as possible to dynamically link)

9. RabbitMQ production line two-time product package (message compensation, Send message persistence, exception handling, monitoring page, duplicate message culling)

1. Designing your message for EDA (event-driven architecture)

Under normal circumstances, when you are using message middleware, it is not designed to be used, and you do not have the application architecture and system architecture boundaries clear. Message middleware is just a purely technical tool that is introduced when you introduce it in the context of application architectures. This is the architectural perspective and the God perspective of the architecture, so you don't use the final discovery to become more chaotic, and you can't combine software patterns, methodologies, and best practices to improve your system's architectural capabilities.

The event-driven architecture of EDA (event driven Architecture,eda) is an architectural pattern used in SOA or in a micro service. There are several advantages to it, flexibility has a high scalability.

(Refer to my SOA Architecture article: SOA Architecture design experience sharing-architecture, responsibility, data consistency)

Since EDA needs to plan how many business entities are within the boundaries of your current system, these entities are built around the domain model. So here's not to be subjective. Define events that you think are designed according to the objects in the business entity. The business entity has at least one identity. For example, orders, commodities, around these entities to expand, the order may have several states are more commonly used, create, pay, distribute, cancel. The product may have price, key attribute modification and so on. The abstraction and refinement of these entities depends on your current business.

(For reference in this context: "Domain driven Design", "exploring Cqrs and event sources")

These are the guiding principles of relative theory, and after that you can land your RABBITMQ so you don't run off. For example, your message name will not look unstructured and layered, deliverymssage (delivery message). It should be, order.delivery.ondeliveryEvented (order. Delivery. Delivery Complete event) such a structure.

When your hierarchy does not meet your business needs, you may also need to further clarify the scope of the event, order.viporder.delivery.ondeliveryEvented (order. VIP order. Delivery. Delivery Complete event).

650) this.width=650; "title=" 1 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 1 "src=" http:// S3.51cto.com/wyfs02/m01/8b/64/wkiom1hl7byjiv6eaablsxgdygg274.png "width=" 855 "height=" 496 "/>

is an event-driven basic scenario, its most noticeable feature is that these, first of all asynchronous, can greatly improve the system's anti-peak ability. Then there is decoupling, which needless to say, the observer pattern in design patterns no one is unaware of its benefits. Scalability, can be scaleout on demand, such as RABBITMQ node can be very convenient to join. The final consistency solves the problem of the CAP theorem in distributed systems.

Design of 2.AMQP Routing key

The design and interaction of routing key is contracted in the AMQP protocol. In order to implement the subscription publishing feature, we need some way to subscribe to events of interest to us. So in the binding of AMQP, pattern matching can be performed according to routing key. So, this can be combined with the AMQP routingkey and domain events, the event that is emitted is equivalent to the Routingkey in AMQP, so it can be perfectly combined.

Your event must be gradually increasing as your business grows, and this collection of events cannot be defined at the outset, so here's one thing to keep in mind: Never write a specific routing key when binding. For example, order.delivery.OnDeliveryEvented, this is the order distribution, at this time when you binding Routingkey is written as "order.delivery.OnDeliveryEvented". The future of order event expansion, it will be very troublesome, unrelated events are subscribed to, can not be refined or events you can not obtain, because Routingkey changed. So when binding, remember the specific point binding, that is, the use of string pattern matching bindings, such as *.delivery.*,*.ondeliveryevented ". More and more routingkey and event will not affect your binding in the future. You just need to be tied to the different levels of the event depending on your level of care.

650) this.width=650; "title=" 2 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 2 "src=" http:// S3.51cto.com/wyfs02/m02/8b/60/wkiol1hl7b2bebbwaaauk1irc8s249.png "width=" 645 "height=" 202 "/>

, Orderbinding binds the order event, which subscribes to the top-level event, which means that any future type of order can be subscribed to, for example, Order.normalorder.delivery.onDeliveryEvent can also be subscribed to. and viporderbinding subscribed to the Viporder event, and if it sends a order.normalorder.delivery.onDeliveryEvent it doesn't matter.

3.RabbitMQ Cluster Construction

Figuring out the application architecture, we started building RABBITMQ cluster. RABBITMQ This AMQP product was developed in Erlang, let's introduce a little bit about Erlang.

The first time I formally contacted Erlang was from RABBITMQ, and at first there was not much to feel special about, but later I realized that the more I realized the more I liked the language. The reason for liking it is that it is a natural distributed language. It sounds like a normal thing to say, but when you get it. The Erlang.cookie mechanism only dawned. An instant epiphany, why use Erlang to make RABBITMQ, but it is really suitable for information exchange and other software. Erlang is a company developed by Ericsson to develop high-performance information switches, and think of those software performance and stability requirements are very high. RABBITMQ node discovery and interconnection is really handy, integrated in Erlang's virtual machines, and highly fault-tolerant. I have a crush on it anyway.

Another thing to be proud of is that RABBITMQ is a great pivotal company, and you should know what pivotal is doing, and if you don't know what it's about, you're not going to get it right now.

650) this.width=650; "title=" 3 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 3 "src=" http:// S3.51cto.com/wyfs02/m00/8b/60/wkiol1hl7b2xnbataaamlurrvnc510.png "width=" 987 "height="/>

At first I did not pay too much attention to their copyright, later on pivotal company more and more admire after suddenly see the original RABBITMQ is their home, suddenly confidence doubled. This is the influence and reputation, look at the company's spring, Springboot, Spring Cloud, admire the pleasantly surprised. (RABBTIMQ official website: http://www.rabbitmq.com/)

3.1. Installing Erlang & RabbitMQ

To install RABBITMQ, you first need to install and configure Erlang for its hosting environment. Go to the Erlang website to download the Erlang OTP_SRC source package, and then perform the source installation locally. (Erlang official website: http://www.erlang.org/)

Since I have downloaded the OTP_SRC source package, I am using the otp_src_19.1 version. After downloading, unzip, then go to the directory, execute./configure--prefix=/usr/erlang/, perform the environment check and installation path selection. If you hint "No curses library functions found" error because of the lack of curses library, yum Install–y ncurses-devel. After installation, the Configure is carried out.

If there is no error, it means the installation was successful. You also need to configure the following environment variables:

Export path= $PATH:/usr/erlang/bin

Source/etc/profile

At this point, use the ERL command to check if Erlang is working properly.

Then install RABBITMQ, go to the official website to download the running package on the line.

Also configure the environment variable so that your command can be found by the system. Then run the RABBITMQ instance.

650) this.width=650; "title=" 4 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 4 "src=" http:// S3.51cto.com/wyfs02/m01/8b/60/wkiol1hl7b6izocbaaarmrrzr_s777.png "width=" 686 "height=" 184 "/>

Here is a need to note, remember to configure the next hosts, in 127.0.0.1 plus the name of the machine. The Erlang process requires host to connect, so it checks your hosts configuration. You also need to set up the next firewall, three ports to open. 15672 is the management interface, 25672 is used between the cluster port, 4369 is the Erlang process EPMD used to do node connection.

I have configured two nodes, 192.168.0.105, 192.168.0.107, now all ready. We added the original account into the RABBITMQ management interface.

3.2. Configure RABBITMQ Cluster

Make sure that your individual RABBITMQ nodes are accessible and open Rabbitmq_management plugin so that you can switch to other management interfaces to view or manage when a node hangs up.

To open the Admin interface plugin:

Rabbitmq-plugins Enable Rabbitmq_management

Add Account:

Rabbitmqctl Add_user Admin Admin

Add Permissions Tag

Rabbitmqctl set_user_tags Admin Administrator

650) this.width=650; "title=" 5 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 5 "src=" http:// S3.51cto.com/wyfs02/m02/8b/60/wkiol1hl7dhtanvcaacdvogdtai450.png "width=" 949 "height=" 521 "/>

650) this.width=650; "title=" 6 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 6 "src=" http:// S3.51cto.com/wyfs02/m02/8b/60/wkiol1hl7dkrd3rhaab-i3urnk0262.png "width=" 944 "height=" 412 "/>

Ensure that all two nodes are working properly. Here we connect the two nodes together to form a highly available cluster so that we can replicate our exchange, queue between the two nodes to form a highly available queue.

CD to your home directory, I am under root, there is a hidden. Erlang.cookie file, which is what I mentioned earlier in this article, which is the basis for Erlang to discover and interconnect. What we need to do is simply to set the. Erlang.cookie in the two nodes to be the same. This is the agreement of Erlang, the same cookie hash key he considers to be legitimate and correct connection.

. Erlang.cookie is read-only by default, you need to modify the Write permission, and then copy and paste the cookie string.

chmod u+w. Erlang.cookie

Once configured, the Hosts file is then configured, and Erlang uses the configuration in the Hosts file to discover the nodes.

Vim/etc/hosts

192.168.0.107 Rabbitmq_node2
192.168.0.105 Rabbitmq_node1

Ensure that the same configuration is the same on all nodes. Verify that your configuration is correct and that you only need to ping rabbitmq_node1 on your machine and try to request IP is not what you configured. According to the principle of DNS request, hosts is the highest priority, unless the browser has a cache, you can directly use ping without problems.

Select a node stop, and then connect to the other node.

Rabbitmqctl Stop_app

Rabbitmqctl Join_cluster [Email protected]_node2

Clustering node [email protected]_node1 with [email protected]_node2 ...

Rabbitmqctl Start_app

The node has successfully connected.

650) this.width=650; "title=" 7 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 7 "src=" http:// S3.51cto.com/wyfs02/m00/8b/64/wkiom1hl7dpbz5xnaacsryy-wt0317.png "width=" 1117 "height=" 434 "/>

By default, nodes occupy 40% of the total memory, and you can study RABBITMQ's configuration items carefully for your own use. To improve performance, no two nodes are required to be disc nodes, so we need to start a node for RAM mode.

Rabbitmqctl Change_cluster_node_type Ram

Change the Rabbitmq_node1 to memory node mode.

650) this.width=650; "title=" 8 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 8 "src=" http:// S3.51cto.com/wyfs02/m01/8b/64/wkiom1hl7dsdjg_faabxzu9leu0032.png "width=" 1179 "height=" 285 "/>

4.Mirror Queue policy settings

The node is ready, and then we need to set up Exchange, queue high-availability policies so that we can really be highly available. It is now a physical machine or a virtual machine node is highly available, but the objects inside require us to configure the policy.

RABBITMQ supports a good policy model and requires an administrator to operate.

First we need to create a vhost within our own business, to mark a logical independent space, all accounts, policies, queues are forced in a virtual machine. I created a common vhost.

Start adding Policie.

650) this.width=650; "title=" 9 "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" 9 "src=" http:// S3.51cto.com/wyfs02/m02/8b/64/wkiom1hl7dwbtyckaabr74_tfpc862.png "width=" 764 "height=" 422 "/>

The most important is apply to, can be in exchange or queues, of course, can also contain these two. The choice of policy is still rich, the most common is Hamode, and Messagettl (the expiration time of the message). These policies are grouped by several dimensions, which are related to high availability, have federation (synchronization messages between clusters), have queue-related, and exchange-related. Adjustments can be made based on the business scenario.

650) this.width=650, "title=", "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" "src="/HTTP/ S3.51cto.com/wyfs02/m00/8b/64/wkiom1hl7datucf7aaassowalfo404.png "width=" 717 "height=" 168 "/>

We define the matching pattern for the policy. order, which avoids mirroring all of the exchange, queue.

650) this.width=650; "title=" one "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" one "src=" http:// S3.51cto.com/wyfs02/m00/8b/60/wkiol1hl7darzrf6aabw8jfrnps408.png "width=" 777 "height=" 288 "/>

We have created a new ex.order.topic exchange, which has a exchange_queue_ha policy applied in its features. (The same policy is not available for stacking.) Other Exchange does not apply this policy because our pattern qualifies only the name of the. Order.

650) this.width=650; "title=" "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" "src=" http:// S3.51cto.com/wyfs02/m01/8b/60/wkiol1hl7dej0a85aaa_mkbdwx0987.png "width=" 1048 "height=" 151 "/>

Create a QU.ORDER.CRM queue and note that it has a "synchronised mirrors:[email protected]_node2" mirror copy in its Node property. The EXCHANGE_QUEUE_HA strategy is also applied in features. At this time, the queue is actually in two nodes, although we created it in the [email protected]_node1, but it will be copied to the other nodes in the cluster. The ha params parameter can be provided when creating hamode to limit the number of replication nodes, which is often used to improve the balance between performance and ha.

5. Two good RABBITMQ plugin large application plug-in (sharding, rederation)

There are two plugin in rabbitmq-plugins that you can try to study. Rabbitmq-plugins list.

Rabbitmq-plugins List
Configured:e = explicitly enabled; E = implicitly enabled
| Status: * = running on [email protected]_node1
|/
[e*] Amqp_client 3.6.5
[] Cowboy 1.0.3
[] Cowlib 1.0.1
[e*] Mochiweb 2.13.1
[] Rabbitmq_amqp1_0 3.6.5
[] Rabbitmq_auth_backend_ldap 3.6.5
[] Rabbitmq_auth_mechanism_ssl 3.6.5
[] Rabbitmq_consistent_hash_exchange 3.6.5
[] Rabbitmq_event_exchange 3.6.5
[] Rabbitmq_federation 3.6.5
[] Rabbitmq_federation_management 3.6.5
[] Rabbitmq_jms_topic_exchange 3.6.5
[e*] Rabbitmq_management 3.6.5
[e*] Rabbitmq_management_agent 3.6.5
[] Rabbitmq_management_visualiser 3.6.5
[] Rabbitmq_mqtt 3.6.5
[] Rabbitmq_recent_history_exchange 1.2.1
[] rabbitmq_sharding 0.1.0
[] Rabbitmq_shovel 3.6.5
[] Rabbitmq_shovel_management 3.6.5
[] Rabbitmq_stomp 3.6.5
[] Rabbitmq_top 3.6.5
[] rabbitmq_tracing 3.6.5
[] Rabbitmq_trust_store 3.6.5
[e*] Rabbitmq_web_dispatch 3.6.5
[] Rabbitmq_web_stomp 3.6.5
[] Rabbitmq_web_stomp_examples 3.6.5
[] Sockjs 0.3.4
[e*] Webmachine 1.10.3

Rabbitmq_sharding, Rabbitmq_federation,rabbitmq_sharding's version is a little low, GitHub address: https://github.com/rabbitmq/rabbitmq-sharding

650) this.width=650; "title=" "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" src= "http:// S3.51cto.com/wyfs02/m01/8b/64/wkiom1hl7djapkaraafwvhaw3us207.png "width=" 644 "height=" 447 "/>

Rederation can be used to synchronize messages across cluster or node. Http://www.rabbitmq.com/federated-exchanges.html

This is a good solution for passing messages between different domain, and subscribing to someone else's RABBITMQ message is always less stable across the computer room or across the network area, and can be used to deliver the message in this way.

650) this.width=650; "title=" "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" "src=" http:// S3.51cto.com/wyfs02/m00/8b/64/wkiom1hl7dqs_et0aad30bniphw591.png "width=" 644 "height=" 471 "/>

6.Queue Mirroring failed manual sync

Sometimes the queue mirror can fail for a variety of reasons, and this time it is possible to synchronize manually instead of restarting the node or rebuilding the data like any other distributed system.

650) this.width=650, "title=", "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" src= "http:// S3.51cto.com/wyfs02/m01/8b/64/wkiom1hl7drxxabzaaaxes8l-kg882.png "width=" "height="/>

650) this.width=650; "title=" border-right-width:0px;background-image:none;border-bottom-width:0px "style="; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" src= "http:// S3.51cto.com/wyfs02/m02/8b/64/wkiom1hl7duyp98maabdtq2jamq512.png "width=" "height=" 242 "/>

This is still more convenient, sometimes there are a few small problems you need to handle manually.

7. Each cluster configuration synchronization mode (RabbitMQ Export\import)

The cluster configuration synchronization of each environment is also a daily operational problem, fortunately RABBITMQ also provides the relevant tools.

650) this.width=650; "title=" "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px; "border=" 0 "alt=" "src="/HTTP// S3.51cto.com/wyfs02/m01/8b/61/wkiol1hl7durmewuaaaviwkgk9o432.png "width=" 613 "height=" 357 "/>

650) this.width=650; "title=" "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" src= "http:// S3.51cto.com/wyfs02/m02/8b/64/wkiom1hl7dyty8hraaaf6khusam271.png "width=" 519 "height=" 269 "/>

650) this.width=650; "title=" style= "border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" src= "http:// S3.51cto.com/wyfs02/m00/8b/64/wkiom1hl7d3rgvioaab36mecahs492.png "width=" 581 "height=" 586 "/>

8. Client connection mode (use AMQP groups as much as possible to dynamically link)

Since RABBITMQ is an implementation of the AMQP protocol, it is possible to connect in the same way as the AMQP protocol when making a remote connection.

var amqplist = new list<amqptcpendpoint>
{
New Amqptcpendpoint (New Uri ("amqp://192.168.0.105:5672")),
New Amqptcpendpoint (New Uri ("amqp://192.168.0.107:5672"))
};

About the cluster VIP solution is also need to consider, if the unified address will face three problems, DNS, LoadBalance, VIP, these three points are likely to cause the cluster connection is not on. More and more scenarios are now prone to load and failover on the client side, which has many benefits, eliminating the failure probability of intermediate nodes. If these three points are added together, the availability metric will certainly be much lower than the direct connection on the client.

We encounter the most is the VIP problem, this kind of system VIP is different from the database, the database master\slave mostly to Manual check after the switch, will not arbitrarily automatically switch master and slave library. The VIP of the non-database is mostly keepalived auto-detect switch, which brings some column problems, including connection retry, Heartbeat hold. This is just one of the VIP's error scenarios. There are also problems with loadbalance, and the likelihood of DNS errors is great. So I tend to use the client to do this.

650) this.width=650; "title=" style= "border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" src= "http:// S3.51cto.com/wyfs02/m00/8b/61/wkiol1hl7d7d7zmyaactuydaloc313.png "width=" 1036 "height=" 618 "/>

A few places are important, the first is the persistent persistence State to take, the second is ContentType, this property is very practical, convenient for you to view the body of the message.

650) this.width=650; "title=" border-right-width:0px;background-image:none;border-bottom-width:0px "style="; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" src= "http:// S3.51cto.com/wyfs02/m01/8b/61/wkiol1hl7d_ryxqaaabvwe31the883.png "width=" 636 "height=" 569 "/>

If not set, the default is null.

The third is automaticrecoveryenabled, automatic connection retry, which is deadly important. When the above VIP switch, this can survive. The fourth one is topologyrecoveryenabled, which restores exchange, queue, and binding. After a network disconnect, these settings are restored once the connection is restored to ensure that the settings are up to date.

9.RabbitMQ production line two-time product package (message compensation, Send message persistence, exception handling, monitoring page, duplicate message culling)

No matter how strong the RABBITMQ is, how high is available, remember that there must be a fallback plan.

Before I wrote an article, WEBAPI's visual output mode (RabbitMQ, message compensation related)--a feature that all WEBAPI seems to be missing

That is the persistence and compensation of the message.

650) this.width=650; "title=" "style=" border-right-width:0px;background-image:none;border-bottom-width:0px; padding-top:0px;padding-left:0px;padding-right:0px;border-top-width:0px, "border=" 0 "alt=" "src=" http:// S3.51cto.com/wyfs02/m02/8b/61/wkiol1hl7eoadavnaaf-e_izplm950.png "width=" 1137 "height=" 543 "/>

Once we've persisted the messages we've sent and received, we're going to do a lot more. Message compensation can be done, not to worry about exceptions. However, when sending a message, it is important to note that the message is persisted before the business logic is processed. In order to handle the monitoring of special activities, it is also possible to develop a certain business to monitor the number of messages received and processed, and then compensate automatically.

In the development of the compensation program, there is a logic is very forgive, when you compensate for a message will be more than send a message, and accept the message must be less than you send. So when you're counting, remember to distinct.

This article from the "humble if foolish, thirst for knowledge" blog, please be sure to keep this source http://wangqingpei557.blog.51cto.com/1009349/1881540

RabbitMQ High Availability cluster construction and e-commerce platform use experience summary