The practice of RABBITMQ in the business system of Yi Long

RABBITMQ, as a mature enterprise message middleware, realizes the decoupling of interface calls between applications and improves the throughput of the system.

Here are some basic concepts of the following RABBITMQ:

• MSG acknowledgment: message acknowledgement, resolving message acknowledgement problem, can not be removed from the message system until ACK is received.
• Message Durability: messages persist, and when RABBITMQ exits or crashes, the messages in the queue are persisted. Note, however, that RABBITMQ does not guarantee that the message must not be lost, because in order to improve performance, RABBITMQ will persist the message in the memory cache until the threshold is reached to the disk, which means that if rabbitmq crashes before persisting to disk, then A small amount of data is lost, which is not unacceptable for most scenarios, and if you do need to ensure that the task is never lost, you should use the transaction mechanism
• Exchange: A mapping relationship that implements a mapping between a message name and a queue and sends messages to the appropriate queue based on the message names.
• Common mapping modes:
• Direct: forwards the message to the queue specified by Routigkey
• Topic: forwarding messages by rules (most flexible)
• Headers
• Fanout: forwarding messages to all bound queues
• Routing: The media that is bound between Exchange and queue becomes routing key

At elong, we developed a RABBITMQ-based messaging system that delivers reliable messaging, provides a simple restful API, and reduces the learning costs of business use RABBITMQ.

The following is the system jmsgThe main component, before saying, is to connect the database structure first:1.MessageConfig send-side configuration, Message->queue mapping relationshipCREATE TABLE ' messageconfig ' (' ID ' int (one) not null auto_increment, ' messagename ' varchar ($) NOT NULL,--Message name' Exchangename ' varchar ($) Not NULL,--mapping relationship of message name and queue' Priority ' varchar (DEFAULT NULL),--the media that was bound before Exchange and queue' Usedelayretry ' bit (1) DEFAULT NULL,-whether to use retry' Delaytime ' int (one) DEFAULT NULL,-How long does the delay retry' Maxretrycount ' int (one) DEFAULT ' 3 ',-Maximum number of retries' _timestamp ' timestamp not NULL DEFAULT current_timestamp on UPDATE current_timestamp, PRIMARY key (' ID '), UNIQUE KEY ' I X_messagename ' (' MessageName ')) Engine=innodb auto_increment=409 DEFAULT Charset=utf8; Field Explanation:messagename: Message nameExchangename:exchange namePriority: A line of business can have multiple queues depending on the priority levelUsedelayretry: Whether to use retryDelaytime: How long does the delay retrymaxretrycount: Maximum retry count table Data 2.MessageConsumersConfig Table: Consumer configuration, Message-and receiver-side configurationCREATE TABLE ' messageconsumersconfig ' (' ID ' int (one) not null auto_increment, ' messagename ' varchar ($) NOT NULL,-Message name' URL ' varchar (+) not NULL,-URL for message consumption' TimeOut ' int (one) default ' Ten ', PRIMARY KEY (' ID ')) Engine=innodb auto_increment=266 default Charset=utf8;messagename: Message nameURL: Message consumption URLTimeout: Consumption time-outCREATE TABLE ' queuesetting ' (' ID ' bigint () not null auto_increment, ' queuename ' varchar () DEFAULT NULL, ' QOS ' int ( One) default null, ' Parallelcount ' int (one) default null, ' LastUpdateTime ' datetime default NULL, ' Lastupdateusername ' VA Rchar () default NULL, PRIMARY KEY (' ID ')) Engine=innodb auto_increment=51 default Charset=utf8;RABBIMQ Configuration

ServerIP: Server IP

PORT: Service port number

UserName: User Name

PassWord: Password

Maxpoolsize: Maximum Connection Pool size

Requestedheartbeat: Request heartbeat Check time (s)

Requestedconnectiontimeout: Request Connection time-out

Failedlogbasedir: Failed log store directory

ConnectionTimeout: Connection hold Time (MS)

Sendtimeout: Send timeout (s)

ReceiveTimeout: Receive time-out (s)

Sendlogbasedir (send log directory)

1 jmsg-client

Message sending client, providing an interface for sending messages

Flow chart:

One of the more important is the Rabbitconnectpool( singleton create Connection Pool ), the more important properties and methods in this class

_max://Maximum number of connections that can be created

_created: Number of connections already in use

_used: Number of connections already in use

_sendtimeout: Send connection request time-out

_receivetimeout: The time-out period for receiving connection success

_clientexpires: Connection Expiry time

_connectiontimeout: Connection timed out

_qos:

Important ways to:

Getsendingconnection (): Gets a connection from a send side, if not enforced, gets the connection from the connection pool, or forces a connection to be created

Getnextproxy: () the past connection from the connection pool (returns Rabbitsendproxy), if the maximum number of connections is exceeded, a new connection is created, otherwise the lock gets proxy (Pollproxy), and if return is empty, wait until the connection is obtained

Pollproxy (): Gets the connection, poll from Proxyqueue, if the connection is not available, this _created–, and then _used++, if the number of bars < maximum number is created, this gets the new connection (Newproxy (), create++, _used+ +

Returntopool (): Returns to the connection pool,

Getnewproxy (): Three retries, GetProxy (), Retry interval 0.1s

GetProxy (): Generating connections from Factory mode

  Rabbitproxy: The client connects the RABBITMQ proxy interface as a unit connection agent in the connection pool, which can be inherited by both the sending and receiving sides

Main properties:

IsAvailable: Available, default True

Createtime: Creation time,

Disposelistener: Connection pooling shutdown interface to execute

ConnectionTimeout: Connection time-out, current time-createtime <= ConnectionTimeout available

ReceiveTimeout: Receive time-out

Connection: The main class, com.rabbitmq.client.Connection connection

QoS: Number of message bars that can be transmitted by the server at one time

Channel: Pipeline, Connection creation pipeline, data transfer

ConnectionFactory: Connection factory, creating RABBITMQ connection

Main operation:

IsAvailable (): Whether the connection is available

Dispose (): Close connection required to close channel and connection   Rabbitsendproxy send-side agent, the default channel Confirsselect, that is, the confirmation mechanism

Send (): Sending method

Process: Convert to byte array, check message length (less than 64K), cache data, wait for acknowledgement, send (basicpublish), delete cached data after receive

Here's how to ensure that the data must be sent to the rabbit queue:

In order to solve the problem of sending failures, the solution is nothing more than the persistence of the message, the use of file persistence is a better choice.

The specific implementation is that after the message failed, put Blockingqueue as the data out of place, periodically read the data store file from the queue, open the timer task read data, re-send to the queue.

2 Jmsg-server

Function: Reads the message from the RABBITMQ, calls the consumer through the HTTP interface

Database

The Jmg-server process:

• Get the queue that the machine needs to process from the database, initialize the RABBITMQ connection pool
• Traverse Queuelist, register listeners, get message handling for each queue
• Turn on the messagereceiver thread for each queue and listen for the queue data
• Messagereceiver Open thread Pool, QoS is thread pool size
• Messagereceiver is a loop that keeps getting RABBITMQ server connections
• After fetching the data, the thread is opened for processing Messageprocesstask, the task is to find the Imessagelistener implementation class, call the Receive method
• Receive Message Processing:

Message verification, get the message configuration, find the consumer---judgment not in process--message has not been processed successfully or failed to reach the maximum number of processing failures –> first received message inbound-> Broadcast message to receiver, processing succeeded, record message Log, modify the state; Processing failed, sent to Rabbitmq-server, waiting for the next processing.

3 Rabbitmq-server

Cluster-based approach to provide a unified URL through Nginx

Some important concepts in the cluster:

Network partition: Networking outage, typically a device outage between subnets, which can cause problems with device communication on different subnets

Build a cluster:

ABBITMQ clusters are dependent on the Erlang cluster to work, so we must first build the cluster of Erlang. The nodes in Erlang's cluster are implemented by a magic cookie in the process, which is stored in the $home/.erlang.cookie (as my root user installs it on my root/. Erlang.cookie), the file is a 400 permission. So it is necessary to ensure that each node cookie is consistent, otherwise the nodes can not communicate.

Scenario 1: Normal cluster

Erlang uses cookies to determine whether it can communicate with another node, and it is common practice to generate cookie files on one machine and copy them to other machines in the cluster.

A cluster can connect multiple machines by means of a single logical broker. Each machine communicates via Erlang messaging, so all nodes within the cluster must have the same Erlang cookie. The network connection between machines in the cluster must be trusted, and all machines must be running the same version of Erlang and RABBITMQ.

Virtual machines, switches, users, and permissions are automatically mirrored to all nodes within the cluster. The queue may be on a single node or mirrored on multiple nodes. The client connects to any node within the cluster to see all the queues within the cluster.

Steps

1

rabbitmq-server-detached rabbitmq-server-detached rabbitmq-server-detached

2 Add to Rabbit3 as the cluster, the cluster name is [email protected], you need to perform the following operations on Rabbit1 and rabbit2, add [email protected],

Rabbitmqctl Stop_app rabbitmqctl join_cluster [email protected] Rabbitmqctl Start_app starting node [email protected] ... done.

3 also in Rabbit3, on the operation, add [email protected]

Rabbitmqctl Stop_app rabbitmqctl join_cluster [email protected] Rabbitmqctl Start_app starting node [email protected] ... done.

Scenario 2: Mirroring queue

The RABBITMQ default cluster mode of the above configuration, but does not guarantee the high availability of the queue, although the Exchange machine, binding these can be copied to any node in the cluster, but the queue content will not replicate, although the mode to solve a project group node pressure, but the queue node downtime directly caused the queue can not be applied, Can only wait for the restart, so in order to be in the queue node downtime or failure can be normal application, it is necessary to copy the queue content to each node in the cluster, the need to create a mirror queue .

Federation allows a switch on one broker to receive messages that are published to a switch on another broker, which may be a separate machine or cluster. To enable communication between nodes through AMQP (with the SSL option), the appropriate users and permissions must be granted between the two switches that make up federation.

A one-way point-to-point connection is made between the switches that make up the federation. By default, on federation connections, messages are forwarded only once, but this can add more and more complex routing topologies.

On a federation connection, some messages may not be forwarded, and if a message cannot be routed to a queue after it reaches the federated switch, it will not be forwarded.

You can pub/sub messages by connecting to each broker on the Internet through Federation.

Solution 3:shovel

Working at a lower level than federation, shovel simply consumes messages from one of the broker's queue and passes to the next broker's exchange

The shovel simply consumes messages from a queue on one broker, and forwards them to an exchange on another.

Resources:

1 http://lynnkong.iteye.com/blog/16996842 http://blog.chinaunix.net/topic/surpershi/

3 http://www.rabbitmq.com/documentation.html

The practice of RABBITMQ in the business system of Yi Long