RABBITMQ cluster, mirrored deployment configuration

1 RABBITMQ Introduction and Installation

RABBITMQ is an open-source AMQP implementation that is written in Erlang and supported by a variety of clients such as Python, Ruby,. NET, Java, JMS, C, PHP, ActionScript, XMPP, stomp, etc., and support Ajax. It is used to store and forward messages in distributed system, which is very good in ease of use, extensibility, high availability and so on.

AMQP, Advanced message Queuing Protocol, is an open standard for application-layer protocols designed for message-oriented middleware. Message middleware is mainly used for decoupling between components, the sender of the message does not need to know the existence of the message consumer, and vice versa.

The main features of AMQP are message-oriented, queue, routing (including point-to-point and publish/subscribe), reliability, and security.

1 RABBITMQ System Architecture

RabbitMQ server: Also known as broker server, is a transport service that maintains a route from producer to consumer to ensure that data is transmitted in the specified manner.

Producer, the sender of the data.

Consumer, the receiver of the data.

Exchanges receives the message, forwards the message to the bound queue. The main use of 3 types: Direct, topic, fanout.

Queue rabbitmq An object that stores messages internally. A queue of the same property can be defined repeatedly, but only if the definition is valid for the first time.

Bindings binds the route between the exchanges and the queue.

Connection: is a TCP connection. Both producer and consumer are connected to RABBITMQ server through TCP.

Channel: Virtual connection. It is established in the above TCP connection. The flow of data is carried out in the channel. In other words, the general scenario is that the program starts with a TCP connection, and the second step is to establish the channel.

2 RABBITMQ installation and start-up

Ubuntu Installation

Edit/etc/apt/sources.list file Add as Line

Deb Http://www.rabbitmq.com/debian/testing Main

Execute Update Software Source command

#apt-get Update.

Installing RABBITMQ

#apt-get Install Rabbitmq-server

Start RABBITMQ

# service Rabbitmq-server Start

Centos Installation

Download the RPM packages for Erlang and RABBITMQ from the official website

Installing Erlang

#rpm –i erlang-17.4-1.el6.x86_64.rpm

Installing RABBITMQ

#rpm –i rabbitmq-server-3.5.3-1.noarch.rpm

RABBITMQ RPM Installation package cannot specify the installation path

3 RABBITMQ configuration file

RABBITMQ Configuration file Location:

• Generic unix-$RABBITMQ _home/etc/rabbitmq/
• debian-/etc/rabbitmq/
• rpm-/etc/rabbitmq/

4 RABBITMQ page (web) management

Enable Web Plugin

#rabbitmq-plugins Enable Rabbitmq_management #启用

Close the Web plugin

# rabbitmq-plugins Disable Rabbitmq_management

You can use the default account guest,guest login http://Host ip:15672, if you want to telnet, you need to create an account, you can view the next section.

5 RABBITMQ Create an account

If it is a cluster, as long as in one host settings, the other will automatically sync.

#rabbitmqctl Add_user IOM 123456–iom for new users, 123456 for passwords

#rabbitmqctl Set_user_tags IOM administrator– to set the user as the Administrator role

#rabbitmqctl set_permissions-p/IOM ". *" ". *" ". *"

– Set up IOM User Configuration permissions, write permissions, read permissions in the/virtual host: * is the usage in regular expressions. The permissions of RABBITMQ are configured according to different virtual hosts, which may not be the same as users in different virtual hosts (Vsan).

2 RABBITMQ Security features 6 publish message acknowledgement mechanism

If the standard AMQP protocol is used, the only way to ensure that the message is not lost is to use the transaction mechanism-to make the channel in transactional mode, to publish the message to it, and to perform a commit action. In this way, the transaction mechanism can incur a significant amount of overhead and result in a 250% decrease in throughput. To remedy the problems caused by the transaction, the confirmation mechanism (Publisher Confirm) was introduced.

The confirm mechanism is to use the Confirm.select method on the channel, the channel in transactional mode can no longer be set to confirm mode, and vice versa.

After the channel is set to confirm mode, all subsequent messages that are publish will be confirm (i.e. ACK) or nack once. But there is no guarantee of the speed of the message being confirm, and the same message will not be confirm and nack.

RabbitMQ will confirm the message in the following scenario:

RABBITMQ found that the current message could not be routed to the specified queues;

Messages of non-persistent properties arrive in all of the queue (and mirror queue) to which they should arrive;

The persistent message arrives in all the queue (and mirror queue) it should reach, and is persisted to disk (Fsync);

Persistent messages are consume from all the queue in which they reside (acknowledge if necessary).

7 Consumer message Acknowledgement mechanism

To ensure that data is not lost, RABBITMQ supports the message acknowledgement mechanism, which is acknowledgments.

If the message acknowledgement mechanism is not started, RABBITMQ will delete the message after consumer receives the message.

After the message acknowledgement is enabled, consumer should send an ACK through the callback function after processing the data, and RABBITMQ will not delete the data until the ACK is received. If consumer does not give back for a period of time, RABBITMQ will reassign the message to another consumer bound on that queue. Another situation is that consumer disconnects, but the obtained message does not give back, and the RABBITMQ is also redistributed.

Note: If consumer does not call the Basic.qos method to set Prefetch_count=1, MESSAGES,RABBITMQ will continue to send messages even if the consumer has an ACK.

8 Persistence of messages

The message acknowledgement mechanism ensures that the message is not lost when the consumer exits, but if the RABBITMQ itself exits with a failure, the message is still lost. To ensure that data is still not lost in the event of an unexpected RABBITMQ, you need to persist both the queue and the message.

Queue persistence: channel.queue_declare (queue= ' Hello ', durable=true)

Message persistence: Channel.basic_publish (exchange= ",

Routing_key= "Task_queue",

Body=message,

Properties=pika. Basicproperties (

Delivery_mode = 2,) #消息持久化

)

Even if there is a message persistence, the data is likely to be lost because RABBITMQ is caching the data first and saving it to the hard disk for certain conditions, during which RABBITMQ unexpected data may be lost.

Online tests have shown that persistence can have a greater impact on the performance of RABBITMQ and may fall by more than 10 times times.

3 RABBITMQ cluster 1 RABBITMQ cluster basic concepts

A RABBITMQ cluster can share user,virtualhosts,queues (open highly Available queues), exchanges, and so on. But the message is only transmitted on the node that was created. When the message enters the A-node queue, consumer pulls from the B-node, RABBITMQ temporarily transmits the message between A and B, takes the message entity in a and sends it to the consumer by B. So consumer should try to connect to each node and fetch messages from it.

RABBITMQ cluster nodes include memory nodes, disk nodes. The memory node's metadata is only placed in memory, and performance is higher than the disk node. However, if message persistence is turned on when message is posted, the performance of memory nodes can only be reflected in resource management, such as adding or removing queues (queue), virtual hosts (vrtual hosts), switches (Exchange), etc. Send and receive message speeds as with disk nodes. A cluster must have at least one disk node.

2 RABBITMQ Build a cluster

Environment: There are three hosts, hostname and IP as follows, RABBITMQ execution user is RABBITMQ, the group belongs to RABBITMQ.

Host name IP

RABBITMQ1 192.168.10.2

RABBITMQ2 192.168.10.3

Rabbitmq3 192.168.10.4

Sync Erlang.cookie

Kill the RABBITMQ process of RABBITMQ2 and RABBITMQ3:

#ps –ef|grep Rab|awk ' {print $} ' |xargs kill-9. – There will be a legacy process with service Rabbitmq-servier stop.

Login Rabbitmq1 (rabbitmq1 on the RABBITMQ service cannot be closed), execute

#cd/var/lib/rabbitmq– into the Erlang.cookie directory, only ls–al can see this file

#chmod 777. erlang*– the file defaults to 400 permissions for easy Transfer, modify permissions first, do not have to operate

#scp. Erlang.cookie [email protected]:/var/lib/rabbitmq– pass this file to another two hosts

#scp. Erlang.cookie [email protected]:/VAR/LIB/RABBITMQ

#chmod. er*– Restore File permissions

Performed on RABBITMQ2 and RABBITMQ3, respectively

#chown RABBITMQ:RABBITMQ. er*– Modify the user and group to which the file belongs

#chmod. er*– Modify File Permissions

#service rabbitmq-server Start

Join the cluster

Query RABBITMQ1 node name

#rabbitmqctl Cluster_status

Cluster Status of node [email protected] ...

[{nodes,[{disc,[[email protected]]}]},{running_nodes,[[email protected] rabbitmq1]}]

... done.

RABBITMQ2 joins the RABBITMQ1 node.

# rabbitmqctl stop_app– turn off RABBITMQ2 service

# rabbitmqctl Join_cluster [email protected]-rabbitmq2 added rabbitmq1, rabbitmq2 must be able to ping through rabbitmq1 hostname.

# rabbitmqctl start_app– Start RABBITMQ2 Service

View cluster information

# Rabbitmqctl cluster_status– should be able to see two nodes at this time. The cluster name is [email protected].

Use the same method to add rabbitmq3 to RABBITMQ1.

Change node Properties

#rabbitmqctl stop_app– Stop RABBITMQ Service

#rabbitmqctl Change_cluster_node_type disc/ram– Change node to disk or memory node

#rabbitmqctl start_app– Open RABBITMQ Service

View cluster status

#rabbitmqctl Cluster_status

[{nodes,[{disc,[[email protected],[email protected],[email protected]}]}, {Running_nodes,[[email protected],[email protected], [email protected]]}]...done.

– The first row is a node member in the cluster, and disc indicates that these are disk nodes.

– The second row is a node member that is running

3 RABBITMQ exit the cluster

Suppose you want to pull rabbitmq2 out of the cluster

Execute on the RABBITMQ2

#rabbitmqctl Stop_app

#rabbitmqctl Reset

#rabbitmqctl Start_app

Execute on the cluster master node

# rabbitmqctl Forget_cluster_node [email protected]

4 RABBITMQ Cluster restart

When the cluster restarts, the last node that is hung up should be the first reboot, if for a special reason (such as a power outage at the same time), without knowing which node is the last one to hang up. The following methods can be used to restart:

Execute on one node first

#rabbitmqctl Force_boot

#service rabbitmq-server Start

Execute on other nodes

#service rabbitmq-server Start

See if the cluster status is normal (to query on all nodes).

#rabbitmqctl Cluster_status

If a node does not join the cluster, you can first exit the cluster and then rejoin the cluster.

The above method is not suitable for memory node restart, the memory node restarts when the disk node to synchronize data, if the disk node is not up, the memory node has failed.

5 Precautions

• The cookie must be exactly the same on all nodes and must be noted when synchronizing.
• Erlang connects the service through the hostname, and must ensure that each host name can be ping-through. You can manually add host names and IP correspondence by editing/etc/hosts. If the hostname ping is not available, the RABBITMQ service startup will fail.
• If the queue is a non-persistent queue, if the node that created the queue fails, the sender and receiver can create the same queue to continue working. However, if the queue is persisted, the service can only continue until the node that created the queue resumes.
• Disk node is required to be online when the cluster metadata is changed, but all disk nodes must be online when the node joins or exits. If you do not exit disk node correctly, the cluster will assume that the node is down and do not join other nodes until the node is restored.

4 RABBITMQ HA1 Mirroring queue Concept

The mirror queue can synchronize the queue and message, and when the primary queue is hung, one from the queue becomes the main queue to replace the work.

The mirror queue is based on the normal cluster mode, so you will have to configure the normal cluster before you can set up the mirror queue.

After the mirroring queue is set up, it will be divided into one master node and multiple slave nodes, and if the primary node goes down, the slave node will have a primary node selected, and the original master node will become slave node.

Although the queue and message exist in all mirror queues, the client reads the data from the primary node regardless of the primary node of the physical surface connection or from the node, and then the primary node synchronizes the state of the queue and message to the slave node. Therefore, multiple clients connecting different mirror queues do not produce the same message that is accepted more than once.

2 Configuring the Mirror queue

In the context of 3.2, we now set the queue named "Hello" to synchronize to all nodes

#rabbitmqctl set_policy ha-all "Hello" ' {"Ha-mode": "All"} '

Ha-all is synchronous mode, which refers to synchronizing to all nodes, there are two other modes ha-exactly means mirroring on a specified number of nodes, the number of nodes is specified by Ha-params, and Ha-nodes is mirrored on the specified node. The node name is specified by Ha-params;

Hello is the queue name for synchronization and can be matched with regular expressions;

{"Ha-mode": "All"} means synchronization to all, the synchronization mode is different, this parameter is also different.

After executing the above command, you can view the queue page in the Web management interface, there will be a +2 tag after the node of the Hello queue, indicating that there are 2 slave nodes, and the master node is currently displayed node (xf7021 is the name of the test, Press 4-2 should be RABBITMQ (1-3)).

5 keepalived and Execute script 1 a-keepalived configuration

Red Word for manual add notes, the original file did not.

VI/ETC/KEEPALIVED/KEEPALIVED.CNF file

Global_defs {

router_id Lvs_master}

Vrrp_instance Vi_1 {

State MASTER

Interface eth0

VIRTUAL_ROUTER_ID 51

Priority 100

Advert_int 1

Authentication {

Auth_type PASS

Auth_pass 1111

}

virtual_ipaddress {

192.168.10.251/24 #rabbitmq

}

}

Virtual_server 192.168.10.251 5672 {

Delay_loop 6

Lb_algo RR

Lb_kind DR

Protocol TCP

Real_server 192.168.10.3 5672 {

Weight 3

Tcp_check {

Connect_timeout 3

Nb_get_retry 3

Delay_before_retry 3

Connect_port 5672

}

}

Real_server 192.168.10.4 5672 {

Weight 3

Tcp_check {

Connect_timeout 3

Nb_get_retry 3

Delay_before_retry 3

Connect_port 5672

}

}

}

2 scripts executed on the RABBITMQ server

lvs_rabbitmq.sh Script content:

#!/bin/bash

vip=192.168.10.251

Case "$" in

Start

Ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP

/sbin/route add-host $VIP Dev lo:0

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

Sysctl-p >/dev/null 2>&1

echo "LVS_VIP server start ok!";;

Stop

Ifconfig lo:0 Down

/sbin/route del $VIP >/dev/null 2>&1

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

echo "LVS_VIP server stoped.";

*)

echo "Arg start|stop."

Exit 1

Esac

Exit 0

6 RABBITMQ Monitoring

There are many RABBITMQ monitoring projects that can be monitored through the web management interface.

There are 4 tabs under the overview page. The main concerns are two totals and nodes.

1 totals

Ready is the amount of messages to be processed, and total is the sum of the messages.

Publish the amount of messages sent per second, deliver the amount of messages received per second.

The following 5 gray long squares represent the corresponding number of module connections respectively.

2 nodes

Name is the node names, the following 5 blue squares represent the number of open files, socket connection number, Erlang processes (temporarily unknown), memory consumption two, disk spare amount; Info displays the node properties, and the mouse on the content will display the corresponding statistics.

7 System Test 3 test environment

Host name IP

vip:192.168.10.251

Client 192.168.10.2– This test send (producer) and receive (consumer) on the same machine

RABBITMQ1 192.168.10.3

RABBITMQ2 192.168.10.4

4 preparatory work

Load Machine Start keepalived

# service Keepalived Start

RABBITMQ1 and RABBITMQ2 execute a 5-2 script

#./lvs_rabbitmq.sh start

Assemble the cluster according to chapters 3rd and 4th, configure the mirror queue, and the node type is best set to the disk node.

Press 第1-5 to create the user.

5 Client Test Scripts

Test the Python language client with RABBITMQ, noting that Python distinguishes the block of statements by indentation. The red part is commented, not on the source.

Send Source:

#vi send.py

#!/usr/bin/env python

Import Pika

Import time

Credentials=pika. Plaincredentials (' IOM ', ' 123456′ ') – Configure the user name and password for the connection

Parameters=pika. Connectionparameters (' 192.168.10.251′,5672, '/', credentials)

Connection=pika. Blockingconnection (Parameters)

Channel=connection.channel ()

Channel.queue_declare (queue= ' Hello ')

Count=0

While count<9999:

Message= ' Hello World ' +str (count)

Count=count+1

Channel.basic_publish (exchange= ", routing_key= ' Hello ', body=message)

Print "Sent%s"% (message)

Time.sleep (1)

Connection.close ()

Receive source code

#vi receive.py

#!/usr/bin/env python

Import Pika

Connection=pika. Blockingconnection (Pika. Connectionparameters (host= ' xf7027′))

Channel=connection.channel ()

Channel.queue_declare (queue= ' Hello ')

print ' [*] waiting for message. To exit Press CTRL + C '

DEF callback (Ch,method,properties,body):

print "[x] Received%r"% (body,)

Channel.basic_consume (callback,queue= ' Hello ', no_ack=true)

Channel.start_consuming ()

6 test process Test keepalived Assignment

Executing on the client

#python send.py

Executing on the load machine

#watch Ipvsadm–ln

You can see that the Activeconn column value of RABBITMQ1 or RABBITMQ2 is 1.

The client re-executes the sender

#python send.py

On the load machine, you can see that the value of the Activeconn column for another RABBITMQ service also becomes 1.

Testing for Disaster tolerance:

The sending and receiving programs are executed separately on the client.

#python send.py

#python receive.py

Then turn off a RABBITMQ node, if it is exactly the client connected to the node, the client sending and receiving program will error exit (the program itself if there is an error in the re-sending mechanism is not affected by any). If the other node is turned off, the program is unaffected.

RABBITMQ cluster, mirrored deployment configuration