Go RABBITMQ Message Queue (a): detailed Introduction Detailed introduction

1. History

RABBITMQ is an open source implementation of the AMQP (Advanced Message Queue) developed by Erlang. The emergence of AMQP in fact is also the needs of the masses of people, although in the world of synchronous messaging communication there are many public standards (such as Cobar IIOP, or SOAP, etc.), but in the asynchronous message processing is not so, only large enterprises have some business implementation (such as Microsoft's MSMQ, IBM's Websphere MQ, etc.), so in the 2006 's June, Cisco, Redhat, Imatix, etc. jointly developed the AMQP open standards.

RABBITMQ is developed and commercially supported by RABBITMQ Technologies Ltd. The company was acquired by SpringSource (a division of VMware) in April 2010. was incorporated into pivotal in May 2013. In fact, Vmware,pivotal and EMC are essentially one. The difference is that VMware is an independent listed subsidiary, and Pivotal is a consolidation of some of EMC's resources and is not available today.

RABBITMQ's official website is http://www.rabbitmq.com.

2. Application Scenarios

Anyway RabbitMQ, or how does AMQP solve the problem, or what is its application scenario?

For a large software system, it will have a lot of components or modules or subsystems or (subsystem or Component or submodule). So how do these modules communicate? There is a big difference between this and the traditional IPC. Many of the traditional IPC are on a single system, the module coupling is very large, not suitable for expansion (Scalability), if the use of sockets so different modules can indeed be deployed to different machines, but there are still many problems to be solved. Like what:

1) How does the sender and receiver of the information maintain this connection, and how is the data lost during this period if the connection to the other party is interrupted?

2) How to reduce the coupling between sender and receiver?

3) How to get priority recipients to receive data first?

4) How do I load balance? Effectively balance the load of the receiver?

5) How to effectively send the data to the relevant recipients? In other words, the receiver subscribe different data, how to do a valid filter.

6) How can it be extensible and even send this communication module to cluster?

7) How to ensure that the recipient receives the complete, correct data?

The AMDQ protocol solves the above problems, and RABBITMQ implements AMQP.

3. System Architecture

Becoming a system architecture may be inappropriate, and may be called a more appropriate system architecture for the application scenario.

This system architecture diagram is copyrighted and belongs to sunjun041640.

RabbitMQ Server: Also known as broker server, it is not a truck that transports food, but a transport service. The exact words are RabbitMQisn ' t a food truck, it's a delivery service. His role is to maintain a route from producer to consumer, ensuring that data can be transmitted in a specified manner. But this guarantee is not a 100% guarantee, but it's enough for normal applications. Of course, for business systems, you can do a layer of data consistency guard, you can completely guarantee the consistency of the system.

Client A & B: Also called producer, the sender of the data. Createmessages and publish (send) them to a broker server (RabbitMQ). A message has two parts: payload (payload) and label (label). Payload as the name implies is the transmitted data. The label is the name of exchange or a tag that describes the payload, and RABBITMQ the label to decide which consumer to send the message to. AMQP only describes the label, and RABBITMQ determines how to use the label's rules.

Client: Also called consumer, the receiver of the data. Consumersattach to a broker server (RabbitMQ) and subscribe to a queue. The queue is likened to a mailbox with a name. When a message arrives at a mailbox, RABBITMQ sends it to one of its subscribers, consumer. Of course, the same message may be sent to many consumer. In this message, only Payload,label has been deleted. For consumer, it is not known who sent this information. Is that the agreement itself is not supported. But of course, if producer sends a payload that contains producer information, it is another matter.

There are three concepts that need to be clarified for the correct transfer of a data from producer to consumer: exchanges, queues, and bindings.

exchanges is where producers publish their messages.

Queuesis where the messages end up and is received by consumers

Bindings is how the messages get routed from the exchange to particular queues.

There are several concepts that are not indicated in the above figure, that is connection (connection), channel (channels, channel).

Connection: is a TCP connection. Both producer and consumer are connected to RABBITMQ server through TCP. As we can see later, the starting point of the program is to establish this TCP connection.

Channels: virtual connection. It is established in the above TCP connection. The flow of data is carried out in the channel. In other words, the general scenario is that the program starts with a TCP connection, and the second step is to establish the channel.

So why use the channel instead of using the TCP connection directly?

For the OS, there is a cost to establishing and shutting down TCP connections, and the frequent establishment of closed TCP connections has a significant impact on the performance of the system, and there are limits to the number of TCP connections, which also limits the ability of the system to handle high concurrency. However, there is no such cost in establishing a channel in a TCP connection. For producer or consumer, multiple channel can be used concurrently for publish or receive. Experiments have shown that 1s of data can publish10k packets. Of course, for different hardware environment, different packet size This data certainly is not the same, but I just want to say, for ordinary consumer or producer, this is enough. If not enough, you should consider how to refine your design for split.

4. Further details clarifying 4.1 using ACK to confirm the correct delivery of the message

By default, if a message has been received correctly by a consumer, the message is removed from the queue. Of course, you can also send the same message to a lot of consumer.

If a queue does not have any consumer Subscribe (subscription), then if the queue has data to arrive, then this data will be the cache, will not be discarded. When a consumer is available, the data is immediately sent to the consumer, which is deleted from the queue when the data is received consumer correctly.

So what is the right to receive it? by ACK. Each message is acknowledged(confirmed, ACK). We can display the ACK in the program, or it can be automatically ack. If there is data not being ACK, then:

RabbitMQ Server will send this information to the next consumer.

If the app has bugs and forgets the ACK, then RABBITMQ server will no longer send data to it because the server considers the consumer processing power Limited.

And the ACK mechanism can act as a limiting function (Benefitto throttling): Sends an ACK after the consumer processing completes the data, or even sends an ACK after an additional delay, which will effectively balance consumer load.

Of course for practical examples, for example, we might merge some data, such as data in the merge 4s, and then sleep 4s before we get the data. Especially in the state of the monitoring system, we do not want all the state to be transmitted in real time, but we want to have a certain delay. This can reduce some IO, and the end user will not feel it.

4.2 Reject a message

There are two ways that the first type of reject can let RABBITMQ server send the message to the next consumer. The second is to delete the message immediately from the queue.

4.3 Creating a queue

Both consumer and Procuder can create a queue through queue.declare . For a channel, consumer cannot declare a queue, but subscribes to another queue. Of course, you can also create a private queue. This allows only the app itself to use this queue. The queue can also be automatically deleted, and the queue labeled Auto-delete will be automatically deleted after the last consumer unsubscribe. So what if we were to create a queue that already existed? Then there will be no effect. Note that there is no effect, that is, the second creation if the parameter is not the same as the first time, the operation succeeds, but the queue's properties are not modified.

So who should be responsible for creating this queue? Is it consumer, or is it producer?

If the queue does not exist, of course consumer will not get any message. But if the queue does not exist, then producer publish's message will be discarded. So, or for data not lost, consumer and producer all try to create the queue! Anyway, this interface doesn't have a problem.

The queue's handling of the load balance is perfect. For multiple consumer, RabbitMQ is sent to different consumer in a balanced Way (round-robin).

4.4 Exchanges

As you can see from the architecture diagram, Procuder Publish's message entered Exchange. Then through the "Routing keys", RABBITMQ will find out which queue the message should be placed in. The queue is also bound by this routing keys.

There are three types of Exchanges:direct, Fanout,topic. Each implements a different routing algorithm (routing algorithm).

· Direct Exchange: If routing key matches, then the message is passed to the appropriate queue. In fact, when the queue is created, it will automatically bind the exchange with the name of the queue as routing key.

· fanout Exchange: The queue broadcast to the response.

· Topic Exchange: pattern matching of keys, such as ab* can be passed to all ab* queue.

4.5 Virtual Hosts

Each virtual host is essentially a RABBITMQ Server, with its own queue,exchagne, and bings rule, and so on. This ensures that you can use RABBITMQ in a number of different application.

Next I'll use Python to illustrate how rabbitmq is used.

Go RABBITMQ Message Queue (a): detailed Introduction Detailed introduction