Detailed description of the Python operation of the RabbitMQ Server Message Queue remote results returned, pythonrabbitmq

Detailed description of the Python operation of the RabbitMQ Server Message Queue remote results returned, pythonrabbitmq

Let's talk about the test environment here: Ubuntu14.04 + Python 2.7.4
RabbitMQ Server

sudo apt-get install rabbitmq-server

Python requires the Pika library to use RabbitMQ

sudo pip install pika

Remote result return
No results are returned after a message is sent from the sender. If you only send a message, there is no problem, but in practice, the receiver often needs to process the received message and return it to the sender.

Processing Method Description: before sending information, the sender generates a temporary queue for receiving messages, which is used to receive returned results. In fact, the concepts of the receiving end and the sending end are vague here, because the sending end also needs to receive messages, and the receiving end also needs to send messages, so here I will use another example to demonstrate this process.

Example content: assume there is a control center and a computing node, the control center will send a natural number N to the computing node, the computing node will add N value 1, and then return to the control center. Here we use center. py to simulate the control center and compute. py to simulate the computing node.

Compute. py code analysis

#! /Usr/bin/env python # coding = utf8import pika # connect to the rabbitmq server connection = pika. blockingConnection (pika. connectionParameters (host = 'localhost') channel = connection. channel () # define the queue channel. queue_declare (queue = 'compute _ queue ') print' [*] Waiting for n' # Add the n value to 1def increase (n ): return n + 1 # define the processing method def request (ch, method, properties, body) for the received message: print "[.] increase (% s) "% (body,) response = increase (int (body) # Send the calculation result back to the control center ch. basic_publish (exchange = '', routing_key = properties. reply_to, body = str (response) ch. basic_ack (delivery_tag = method. delivery_tag) channel. basic_qos (prefetch_count = 1) channel. basic_consume (request, queue = 'compute _ queue ') channel. start_consuming ()

The code of the computing node is relatively simple. It is worth mentioning that the original method of receiving messages is to print out the messages directly. Here, we perform the plus-one calculation and send the results back to the control center.

Center. py code analysis

#! /Usr/bin/env python # coding = utf8import pika class Center (object): def _ init _ (self): self. connection = pika. blockingConnection (pika. connectionParameters (host = 'localhost') self. channel = self. connection. channel () # define the queue result = self. channel. queue_declare (exclusive = True) self. callback_queue = result. method. queue self. channel. basic_consume (self. on_response, no_ack = True, queue = self. callback_queue) # define the processing method def on_response (self, ch, method, props, body) for receiving the returned message: self. response = body def request (self, n): self. response = None # Send the computing request and declare that the message is returned to the queue self. channel. basic_publish (exchange = '', routing_key = 'compute _ queue ', properties = pika. basicProperties (reply_to = self. callback_queue,), body = str (n) # receives the returned data while self. response is None: self. connection. process_data_events () return int (self. response) center = Center () print "[x] Requesting increase (30)" response = center. request (30) print "[.] got % r "% (response ,)

The code above defines the queue and processing method for receiving returned data, and assigns this queue to reply_to when sending a request, in the compute Node Code, this parameter is used to obtain the return queue.

Open two terminals, one running code python compute. py, And the other running center. py. If the execution is successful, you can see the effect.

During the test, I encountered some minor problems, that is, in the center. when py sends a message, it does not specify the returned queue. The result is compute. when py is over, an error is returned, indicating that the routing_key does not exist and an error is returned. View the queue with rabbitmqctl list_queues and find that the compute_queue queue has one piece of data. This data will be reprocessed every time compute. py is re-run. Later, use/etc/init. d/rabbitmq-server restart to restart rabbitmq.

Correlation id
The previous example of remote results returned is demonstrated, but one of them is not mentioned, that is, correlation id. What is this?

If there are multiple computing nodes, the control center starts multiple threads and sends numbers to these computing nodes. The calculation results are required and returned, but the control center only opens one queue, all threads get messages from this queue. How can each thread determine that the received message is corresponding to this thread? This is the use of correlation id. When correlation is translated into Chinese, it is related to each other.

How correlation id works: when the control center sends a computing request, it sets the correlation id. Then, the computing node returns the computing result together with the received correlation id, in this way, the control center can identify the request through correlation id. In fact, the correlation id can also be understood as the unique identifier of the request.

Example content: multiple threads are enabled in the control center, and each thread initiates a computing request. With the correlation id, each thread can receive the corresponding computing result accurately.

Compute. py code analysis

Compared with the previous article, you only need to modify one place: When the calculation result is sent back to the control center, the setting of the correlation_id parameter is added. The value of this parameter is actually sent from the control center, here we will send it back again. The Code is as follows:

#! /Usr/bin/env python # coding = utf8import pika # connect to the rabbitmq server connection = pika. blockingConnection (pika. connectionParameters (host = 'localhost') channel = connection. channel () # define the queue channel. queue_declare (queue = 'compute _ queue ') print' [*] Waiting for n' # Add the n value to 1def increase (n ): return n + 1 # define the processing method def request (ch, method, props, body) for the received message: print "[.] increase (% s) "% (body,) response = increase (int (body) # Send the calculation result back to the control center and add the correlation_id setting ch. basic_publish (exchange = '', routing_key = props. reply_to, properties = pika. basicProperties (correlation_id = \ props. correlation_id), body = str (response) ch. basic_ack (delivery_tag = method. delivery_tag) channel. basic_qos (prefetch_count = 1) channel. basic_consume (request, queue = 'compute _ queue ') channel. start_consuming ()

Center. py code analysis

The code of the Control Center is a little more complex. Three of the key points are as follows:

Use the python uuid to generate a unique correlation_id.
Set the parameter correlation_id when sending a computing request.
Define a dictionary to save the returned data, and the key value is the correlation_id generated by the corresponding thread.
The Code is as follows:

#! /Usr/bin/env python # coding = utf8import pika, threading, uuid # custom Thread class, inheriting threading. threadclass MyThread (threading. thread): def _ init _ (self, func, num): super (MyThread, self ). _ init _ () self. func = func self. num = num def run (self): print "[x] Requesting increase (% d)" % self. num response = self. func (self. num) print "[.] increase (% d) = % d "% (self. num, response) # control Center class Center (object): def _ init _ (self): self. connection = pika. blockingConnection (pika. connectionParameters (host = 'localhost') self. channel = self. connection. channel () # define the queue result = self. channel. queue_declare (exclusive = True) self. callback_queue = result. method. queue self. channel. basic_consume (self. on_response, no_ack = True, queue = self. callback_queue) # The returned results are stored in the dictionary self. response = {}# define the processing method def on_response (self, ch, method, props, body) for receiving the returned message: self. response [props. correlation_id] = body def request (self, n): Response _id = str (uuid. uuid4 () self. response [response _id] = None # Send a computing request and set the return queue and correlation_id self. channel. basic_publish (exchange = '', routing_key = 'compute _ queue ', properties = pika. basicProperties (reply_to = self. callback_queue, correlation_id = Response _id,), body = str (n) # receive the returned data while self. response [response _id] is None: self. connection. process_data_events () return int (self. response [calculate _id]) center = Center () # initiate 5 computing requests nums = [10, 20, 30, 40, 50] threads = [] for num in nums: threads. append (MyThread (center. request, num) for thread in threads: thread. start () for thread in threads: thread. join ()

I started two terminals to run compute. py and one terminal to run center. py. The output is as follows:

We can see that although the obtained results are not output in sequence, the results and source data correspond to each other.

This example creates a queue and uses the correlation id to identify each request. You can also choose not to use the correlation id, that is, create a temporary queue for each request. However, this consumes too much performance and is not recommended by the official authorities.