RabbitMQ-high-availability cluster deployment practices

RabbitMQ-high-availability cluster deployment practices

In the previous chapters, we talked about setting up and using the single-Host Mode of RabbitMQ. In the actual production environment, we will use the cluster mode to deploy RabbitMQ for performance and availability considerations.

Basic concepts of RabbitMQ Clusters

The Rabbit mode can be divided into the following three modes: Single-host mode, common cluster mode, and image cluster mode.

Single Host Mode:

The RabbitMQ service runs on a single host. This mode is usually not used in the production environment and has limited performance. In addition, if the server goes down, the service will be completely unavailable.

Common cluster mode

The cluster problems become much more complicated. For the Queue, the message entity only exists on one of the nodes, and other nodes in the cluster only have the same metadata, that is, the Queue structure.

When A message enters the Queue of node A and Consumer pulls the message from Node B, RabbitMQ will temporarily transmit the message between the two nodes, extract the message entity in A and send it to Consumer after B. Therefore, Consumer should try to connect to every node and retrieve messages from it. That is, for the same logical Queue, a Queue must be created on multiple nodes. Otherwise, if the Consumer connects to only one node area, the performance bottleneck of the node will be caused.

This mode has a problem that when one node fails, other nodes cannot receive messages that have not been consumed by the faulty node. If message persistence is performed, you must wait until node A is restored before consumption can be made. If no persistence is available, you can get A cup of cake!

Image Cluster Mode

As mentioned above, in the normal cluster mode of RabbitMQ, different nodes only synchronize messages in the queue structure. In image mode, the queue structure and messages are stored in multiple nodes and belong to the HA solution of RabbitMQ. The essence of the difference is that the message entity will actively synchronize between mirror nodes. This mode also has obvious side effects. In addition to reducing system performance, if there are too many mirror queues and a large number of messages enter, the network bandwidth inside the cluster will be greatly consumed by such synchronous communication. Therefore, this mode is applied to scenarios with high reliability requirements.

Memory nodes and disk nodes

RabbitMQ cluster nodes include Memory nodes and disk nodes. As the name implies, a memory node places all data in the memory, and a disk node places data in the disk. However, as mentioned above, if message persistence is enabled during message delivery, data will still be stored on disks even on Memory nodes. In principle, a cluster must have at least one disk node. In actual use, it is found that the so-called disk node is only used to store the cluster configuration information. That is to say, if there is no disk node in the cluster, the cluster configuration information will be lost after all nodes are shut down. There is no big difference in the performance of two nodes that subscribe to and publish messages during performance tests.

Multi-node load distribution

The RabbitMQ cluster mode has no central node, and the Consumer actually connects to one of the nodes when connecting to the cluster. The connection method is consistent with the single host mode. Then there is an embarrassing problem. How can we ensure that the Consumer is evenly connected to multiple nodes. The following are some of my thoughts and two ideas are provided.

1. Use Server Load balancer devices to distribute traffic. You can use F5 hardware Server Load balancer. If you do not have F5 hardware Server Load balancer devices, you can also use services such as LVS. When a Consumer connects to a cluster, it actually goes through load balancing first.

2. Although Server Load balancer devices are usually stable, the RabbitMQ cluster has a central node. This is what we use. First, we put the IP addresses of nodes in the cluster in an array. When the app connects to RabbitMQ, it randomly selects an IP address from the array to connect, then, the IP address of the connected node is cached to the server. If the connection times out, other nodes are randomly selected for connection. This method is used to distribute app traffic.

Now I have an understanding of the basic concepts of clusters. Let's build a common cluster together.

RabbitMQ cluster deployment

I use five servers to build a cluster with five nodes, of which 10.99.121.150 is a disk node and other servers are Memory nodes. The name of the server is as follows:

10.99.121.150 RMQ_D_150
10.99.121.151 RMQ_M_151
10.99.121.152 RMQ_M_152
10.99.121.153 RMQ_M_153
10.99.121.154 RMQ_M_154

First, install the RabbitMQ of five single hosts.

Modify the host file of each host:

vi /etc/hosts    10.99.121.150 RMQ_D_150    10.99.121.151 RMQ_M_151    10.99.121.152 RMQ_M_152    10.99.121.153 RMQ_M_153    10.99.121.154 RMQ_M_154

Modify the host names of each host: (I did not list them one by one)

vi /etc/sysconfig/network    NETWORKING=yes    NETWORKING_IPV6=no    HOSTNAME= RMQ-M-154

Open the corresponding port of each host:

firewall-cmd --permanent --add-port=25672/tcpfirewall-cmd --permanent --add-port=15672/tcpfirewall-cmd --permanent --add-port=5672/tcpfirewall-cmd --permanent --add-port=4369/tcpsystemctl restart firewalld.service

Synchronize each node Cookie (executed at 150)
Rabbitmq clusters rely on erlang clusters. Therefore, you must first build an erlang cluster environment. Each node in the Erlang cluster is implemented through a magic cookie. This cookie is stored in/var/lib/rabbitmq/. erlang. cookie, and the file has 400 permissions. Therefore, the cookies of each node must be consistent; otherwise, the communication between nodes fails.

scp /root/.erlang.cookie root@10.99.121.151:/root/ scp /root/.erlang.cookie root@10.99.121.152:/root/scp /root/.erlang.cookie root@10.99.121.153:/root/scp /root/.erlang.cookie root@10.99.121.154:/root/

Restart the node.

rabbitmqctl stoprabbitmq-server -detached

Connect to a cluster

Rabbitmqctl stop_app (do not execute the hard disk node first) rabbitmqctl join_cluster -- ram rabbit @ RMQ_D_150 (connect to any node that has already joined the cluster) rabbitmqctl start_apprabbitmqctl cluster_status // view the cluster status // disk node. Run the join_cluster command to remove the -- ram parameter. // In A RabbitMQ cluster, at least one disk node must exist (the disk node is used to store the cluster status ).

Remote access configuration
After configuring the cluster, you need to add a new account
By default, webpage access is not allowed. You need to add a user to modify the permission. The Code is as follows:

Rabbitmqctl add_user superrd // Add User rabbitmqctl set_permissions-p/superrd ". *" // Add permission rabbitmqctl set_user_tags superrd administrator // modify user role

Enter the WEB management plug-in http: // ip: 15672 for any node in the browser. On the Overviem view, find the Nodes tab to view all Nodes.

Delete a node:
Modify the host and host name.
Run the following command on the node to be detached from the cluster:

rabbitmqctl  stop_apprabbitmqctl  restrabbitmqctl  start_app

Or run the following command on another node: (for example, delete the RMQ_M_154 node)

rabbitmqctl stop_apprabbitmqctl forget_cluster_node rabbit@RMQ_M_154

Add nodes:
Copy cookies on existing nodes to new nodes

Scp/root/. erlang. cookie root@10.99.121.155:/root // execute on an existing node. // Run the following command on the node to be added. Rabbitmqctl stop_apprabbitmqctl join_cluster -- ram rabbit @ RMQ_M_154rabbitmqctl start_app

View the cluster status:

rabbitmqctl cluster_status

RabbitMQ image cluster configuration

Although we have deployed clusters in normal mode, because only the queue structure is synchronized between nodes, messages are not synchronized, for scenarios with high reliability requirements, messages in the queue must be synchronized to the nodes.
The Rabbit image function must be implemented based on the rabbitmq Policy. The policy is used to control and modify the behavior of a vhost queue and Exchange in the cluster range, and to enable the policy on any node in the cluster, the policy is automatically synchronized to the cluster node.

You can use commands or WEB to modify the policy. It is very simple if I modify the policy through WEB.
Pattern: "^" indicates all names matching all queues ." ^ Log refers to the name of the queue starting with "log" synchronization.
Ha-mode: "all" indicates synchronization to the node.

After entering the information, click "Add policy" to apply the Configuration policy. You can see the created policy.

Create a new queue and view the queue list. A "+ 4" indicates that the data is saved in four copies.

Click queue to view queue details. We can see that the queue was created on node 150 but synchronized to the remaining four nodes.