RABBITMQ Basic article (i)

This series of blog mainly from the practice of the theory of two points of view with everyone step-by-step understanding of the familiar rabbitmq, a total of basic, advanced, high-level article. The basic article mainly introduces some basic concepts of MQ, RabbitMQ's historical background, and some of the same types of technology, build a running environment and write a simple program, advanced article we will use RabbitMQ do more experiments, in a practical way in-depth study, high-level chapter will explain some RabbitMQ Advanced applications, how to build RabbitMQ's high-availability infrastructure and RabbitMQ integrated plug-ins, and summarize RabbitMQ's design ideas. Why use MQ

MQ allows our requests and processing to be separated, the client's request server can be executed asynchronously, the completion of the return of the results, this decoupling greatly improves the server's ability to respond to client requests, the asynchronous request reduces the number of requests processed by the server at the same time, It is very helpful for us to build high-performance services. What is RabbitMQ

keyword Explanation:
0. AMQP protocol:
The amqp:advanced message Queuing Protocol Advanced Messaging Queuing protocol is an application-layer protocol specification used by asynchronous messaging.

1. ErLang:
Erlang is a general-purpose parallel programming language developed by Chow Armstrong Joe Armstrong in the Computer Science Laboratory of the Swedish telecommunications equipment manufacturer, Ericsson , to create a programming language and operating environment that can handle large-scale development activities. Erlang released its official version in 1987, the first proprietary software owned by Ericsson, and, after 10 years of development, published an open source version in 1998.
Erlang is an interpreted language that operates on virtual machines , but now contains native code compilers developed by the Uppsala University High Performance Erlang Program (HiPE) [2], which is also supported in scripting, starting with the r11b-4 version. In the programming paradigm, Erlang is a multi-paradigm programming language that covers functional, parallel, and distributed. Erlang, which runs sequentially, is a functional programming language for early evaluation, single assignment, and dynamic typing.

Language Features:
Cross-process communication:
In a parallel programming language, you can use the Spawn function to set a specific function as a separate process and then cross-process communication.
Function-Oriented programming
Since Erlang was developed early in Prolog, it became a functional language influenced by linguistic characteristics.
Fast failure
Errors that occur during runtime are submitted by the wrong location, and the process where the error occurs immediately stops executing. By the process communication mechanism, you can automatically pass errors, catch errors, and enable other processes to help handle errors.

2. Producers:
The generator and sender of the message is the starting point for the entire message lifecycle, creating a Message Settings label, and then establishing a connection to the RABBITMQ Proxy server (Erlang node) and sending a message to the corresponding server.

3. Consumers:
The recipient and consumer of the message, receiving a message from the RABBITMQ proxy, parsing the request and responding based on the content, noting that the consumer receives only part of the message: payload, the label of the message is not passed along with the payload during message routing Each message received by the consumer needs to be confirmed, and only after confirming that RabbitMQ can safely delete the data on the server and send a new message to the consumer.

4. Message:
The message consists of two parts: payload (payload) and label, payload is the specific data content that the producer needs to transmit, the label is the description of the data, the equivalent of the metadata, the message is sent in the process of only payload to propagate, the label will not be sent to the consumer.

5. Queue:
The queue is equivalent to a mailbox server named Mailbox, the RABBITMQ server is equivalent to a post office, a post office has multiple mailboxes, a mailbox for multiple recipients, the recipient will always check the mailbox letter, the recipient corresponding to the consumer here, the consumer subscription queue, and constantly receive messages from the queue for processing, When there are multiple subscribers on a queue, messages are distributed to different consumers in a circular (round-robin) fashion. Provides the premises for the message, where the message waits for consumption. For easy load balancing, a queue can correspond to multiple consumers and distribute requests to different consumers in a circular manner. The queue is the endpoint of the message in Rabbit.

6. Switching device:
Similar to routers, complete message distribution, a switch may bind multiple queues, multiple switches may also bind to the same queue, and the exchanger will route messages to the corresponding queue based on route keys (routing key). There are four types of exchangers: direct: If the routing key matches, the message is posted to the corresponding queue. There is a default switch on the server, which is bound to the default switch when a queue queue_name is created, and the name of the queue is used as the routing key. The default switch + temporary queue ==> RPC traffic . Fanout: Broadcasts the received message to the bound queue, and the producer's code is completely decoupled from the consumer's code. Topic: Allows messages from different sources to reach the same queue (log system). Headers: (not commonly used)
The direct switch is a one -to-one mode, the fanout switch is a broadcast mode, a one-to-many mode, the topic switch is a multi-pair mode, each queue has a specific theme, Messages from different applications are routed to the same message queue through the routing key.

7. Binding:
There is a binding relationship between the exchanger (Exchange) and the queue, which is accomplished primarily through two commands: Basic.publish (body, Exchange, Routing_key), Queue.bind (queue, Exchange, Routing_key).
For example: Basic.publish ("Log", "Logs_exchange", "Error.log"), Queue_bind ("Msg_log_queue", "Logs_exchange", "error."), With such a binding, Logs_exchange all error above the exchanger. Logs are sent to the Msg_log_queue queue.

8. Virtual Host:
A RABBIMTMQ server can create multiple virtual host vhost, each of which is isolated from each other, each with its own exchanger, queue, and bindings. After installing RabbitMQ, the default vhost is generated: "/", the default user name password is GUEST/GUEST,RABBITMQ permissions control is in vhost units, vhost on different users set different permissions. Using Vhost can guarantee the security and portability of the communication architecture, and the entire cluster will create the vhost when the Vhost is created on the cluster.
So how to create it: complete the creation through the client tool RABBITMQCTL.
rabbitmqctl Add_vhost[vhost_name]: Create Vhost
rabbitmqctl Delete_vhost[vhost_name]: Delete vhost
rabbitmqctl list_vhosts: list all Vhost

9. Persistence:
Implementation: The message delivery mode (dilivery mode) option is set to 2 (persistent): message is marked as persisted to the persisted queue on the switch that is sent to the persistence
three conditions are indispensable.

Principle:
A persistent log file is created on the disk, and if the message is sent to a non-persisted queue, the message is deleted, if the persisted message is consumed, the message in the persistence log is marked for garbage collection, the server restarts, and the messages in the persisted log file are replayed. So the messages recorded in the persistent log file are persistent messages, which can be restored after the server restarts, so it is common to set some important messages to persist, but because of the need for constant disk reads and writes, performance is not good and reduces RabbitMQ message throughput. It is possible to reduce performance by as much as 10 times times, and to work poorly in a built-in cluster environment (because the built-in cluster data is not redundant)

10. Strategy:
(1) Persistent transaction: The AMQP transaction differs from the database transaction, and multiple commands form a transaction, and only the first command executes the other commands. synchronous, poor performance.
(2) Sender acknowledgement mode (confirm mode): Messages posted on all channels are assigned a unique ID number (starting from 1), once the message is posted to all matching queues, the channel sends a sender acknowledgement mode to The producer Application (which contains the unique ID of the message) so that the producer knows that the message has been sent successfully, and if both the message and the queue are persisted, then the acknowledgment message is sent after the queue writes the message to disk, and the greatest advantage of the sender acknowledgement mode is that it is asynchronous. The producer can send the next message at the same time as the confirmation message, and if it receives a nack (unacknowledged) message, the message is re-sent.

11, Black Hole: The route message can not find the corresponding queue, then the message will enter the "black hole."

12. Dead Letter: "garbage (Exception) message" Rejected by the consumer and not re-queued, stored in the dead letter queue.

Common commands:

Basic.get: requests a single message to the queue for consumption.

basic.ack: The consumer explicitly confirms that the message has been received.

basic.reject: An error occurred during the consumption process, the current server could not process the message, an explicit refusal to continue sending, and the Requeue parameter set to True, RABBITMQ would distribute the message to consumers of other subscriptions. When the Requeue parameter is set to False, RabbitMQ removes the message from the queue, which enters "dead letter" (the message that holds the rejected and cannot be re-queued).
Queue.declare: To create a queue, you typically specify a name and do not specify that it is randomly assigned to create a "temporary anonymous" queue. Exclusive: Set to true to indicate a private queue. Auto-delete: The last time a consumer cancels a subscription, it is automatically deleted. Passive: The test queue exists, there is a successful return, and there is no return error.

basic.publish (body, Exchange, Routing_key): messages are sent, sent from the exchanger to the corresponding queue, and MSG messages are sent to the corresponding queue via the default switch. Send a message to specify the message body, the target switch, and the queue on the corresponding switch.

$channel-Basic.publish ($msg, ', ' queue_name ')

exchange.declare: Creating a switch

queue.bind(queue, Exchange, Routing_key): Exchanger Queue Binding

Basic.consume (Msg_consumer, queue, Consumer_tag): The consumer subscription queue, once subscribed, will continue to monitor the queue, consume (reject) the end of a line and continue to receive the next article. Msg_consumer (channel, method, header, body): The message processor, Basic.consume receives the message with the handler to process the message, channel is the message source channels, method It contains some details of the sender of the message, such as the exchanger name (exchanges), Route keys (Routing_key), the delivery mode (Delivery_tag), and so on, the header is the message header, body is the message body. Queue: The name of the queues that the consumer subscribes to. Consumer_tag: The unique identity of the consumer.

basic.cancel: Unsubscribe, consumers no longer subscribe to the queue

start_consuming: Consumers start listening to queues

stop_consuming: Consumer stops listening queue

General consumer processes:
Basic.consume (subscription) –> start_consuming (start listening) –> Basic.cancel (unsubscribe) –> stop_consuming (stop listening) environment Setup

(1) Download and install Rabbitmq:rabbitmq-server-3.6.9.exe
(2) Configuring the boot server
To configure environment variables:
D:\Otp\erl8.3\bin Add to environment variable path inside

Start the server:
CD to RABBITMQ's Sbin directory
rabbitmq-server.bat start: start RABBITMQ server: 5672 RABBITMQ, 15672 client Plug-in, 25,672 cluster, 4369 EPMD

To view the startup status:
rabbitmqctl.bat Status: View Erlang node status
rabbitmq-plugins.bat Enable rabbitmq_management: Allow plugin access
Access path: http://localhost:15672/

Create a user and authorize
rabbitmqctl.bat add_user lijl85 test: Create user
rabbitmqctl.bat set_permissions "." “.” ". *": authorization, three parameters corresponding to the configuration, read, write permissions.

(3) Client access:
Python access: Using Pip to install Pikachu (PIKA:RABBITMQ python implementation)

/
 * * Introduce Pika */
 import JSON, Pika

Hello world!

# coding = utf-8 # Broker_util.py:get_channel can get a proxy server channel import Pika # get Proxy server channel def Get_chan Nel (vhost, server, user, password): Amqp_vhost = vhost Amqp_server = Server Amqp_user = user Amqp_pass = P Assword Credentials_broker = Pika. Plaincredentials (Amqp_user, amqp_pass) Conn_params = Pika. Connectionparameters (Credentials=credentials_broker, Virtual_host=amqp_vhost, host=amqp_server) connection = Pika. Blockingconnection (Conn_params) return Connection.channel () 

# producer.py: producer, issue Hello World message Import JSON, Pika import common.broker_util as broker; Channel = Broker.get_channel (vhost= "/", server= "localhost", user= "lijl85", password= "test") channel.exchange_declare (exchange= "Hello_exchange", type= "direct", Auto_delete=false) Channel.queue_declare (queue= "Hello_queue", auto_ Delete=false) Channel.queue_bind (queue= "Hello_queue", exchange= "Hello_exchange", routing_key= "Hello") msg = Json.dumps ({"Data": "Hello RabbitMQ world!"}) Msg_props = Pika. Basicproperties () Msg_props.content_type = "Application/json" msg_props.durable = False # non-persisted message channel.basic_publish ( Properties=msg_props, Body=msg, exchange= "Hello_exchange", routing_key= "Hello") 

# consumer.py: Consumer

import JSON
import common.broker_util as broker;

Channel = Broker.get_channel (vhost= "/", server= "localhost", user= "lijl85", password= "test")
Channel.exchange_ Declare (exchange= "Hello_exchange", type= "direct", Auto_delete=false)
channel.queue_declare (queue= "Hello_ Queue ", Auto_delete=false)
channel.queue_bind (queue=" Hello_queue ", exchange=" Hello_exchange ", routing_key=" Hello ")

def msg_handler (Hello_channel, method, Hello_header, body):
    msg = json.loads (body)
    print (msg.get ("Data"))
    Channel.basic_ack (Delivery_tag=method.delivery_tag)

Channel.basic_consume (Consumer_callback=msg_handler, Queue= "Hello_queue", No_ack=false, consumer_tag= "Hello_consumer")
channel.start_consuming ()

Broker_util is a separate encapsulated module for obtaining a channel with a proxy server that needs to pass in the currently used Vhost, server, user, password, username password can be set in RabbitMQ Management Web UI, When you use it, start consumer.py, and then execute producer.py to see a message sent.

Execution Result:

User Configuration:

Exchanger Configuration:

Queue configuration:

message send panel:

By logging in to http://localhost:15672 we can see the various information about the Erlang node and the message sent, RabbitMQ the Web UI is more powerful than the Rabbitmqctl tool is that it can see the classification of the message: The message to be sent + confirmed message + unacknowledged message + Total Message Bar count. Intuitive and friendly.