1. IntroductionIn Hello World, have learned how to send and receive messages, but in the actual application process, not simply receive and send. For example: When we have complex needs, we need to improve efficiency, after all, only one consumer can not handle it, as mentioned in the official website--In this tutorial, you will create a Task Force column (Work Queue), which will send some time-consuming tasks to multiple workers (worker). Work queues (also known as task Queues--task queues) are designed to avoid waiting for operations that take up a lot of resources and time. When we send a task as a message to a queue, a worker process running in the background takes out the task and processes it. When you run multiple workers (workers), the tasks are shared among them.
2. Application ScenariosIn Hello World just send a string alone, but the actual application may be pictures, PDF processing, etc., may need to consume a different length of time, according to the official website example does not do complex operations, but use the symbol "." To represent the ease of the task, thereby controlling the duration of the task with Time.sleep (). one point (.) will take 1 seconds. For example, "Hello ..." can take 3 seconds.
3. Work QueuesWe need to modify on the basis of Hello World, combined with the previous chapter of the explanation of the parameters, one by one processing, we first introduce the following key words, and finally the summary and integration.
Cyclic schedulingOne advantage of working with a work queue is that it can process queues in parallel. If a lot of tasks are piled up, we just need to add more workers (workers), and then the extensions are simple. By default, RABBITMQ sends messages to each consumer (consumer) sequentially. On average, every consumer receives the same amount of information. This way of sending messages is called-polling (Round-robin).
Message ConfirmationIn short, when rabbitmq the task to the worker (i.e., the consumer), only when the worker returns a confirmation message to RABBITMQ, RABBITMQ thinks the message is handled correctly and then removes the message. Otherwise, RABBITMQ will continue to hold the message without processing, and when the worker hangs up, RABBITMQ will hand over the task to another worker to process it until the processing is complete. Based on this, RABBITMQ provides a message response (acknowledgments). The consumer will pass an ACK (response) telling RABBITMQ that a message has been received and processed, and then RABBITMQ will release and delete the message. The message is not timed out; RABBITMQ will resend the message when the worker is disconnected from it. This will not be a problem when dealing with a very long message task.
Message response is turned on by default. In the previous example we can use the No_ack=true logo to turn it off. We need to remove this identity below and send a response after the worker (worker) completes the task.
The code is as follows:
DEF callback (ch, method, properties, body): "" "
@ch: Channel channel, is to be sent out from the outside when the body
read the
contents of the queue to do different operations
" '
print ' [x] recived%r '% (body,)
print ' [x] ch {0} '. Format (CH)
Print [x] method {0} '. Format (method)
print ' [x] Properties {0} '. Format (properties)
Time.sleep (Body.count ('. '
)) print ' [x] done%r '% (body,)
Ch.basic_ack (Delivery_tag=method.delivery_tag)
Use Basic_ack to send a confirmation message.
Note:Forgetting to confirm is also a very dangerous thing. Technically, this message will not be removed from memory even if it is processed, which will result in less usable memory and eventually a memory overflow. From the business point of view, the same message was processed two times, involving the business model may cause duplication of processing, causing serious consequences. For this problem, you can also output messages_unacknowledged fields by Rabbitmqctl:
[Trabbitmq@iz250x18mnzz src]$ sudo rabbitmqctl list_queues name Messages_ready messages_unacknowledged Listing
queues ... Task_queue 0 0
Hello 0 0
Persistence ofIf you don't tell RABBITMQ specifically, all queues and messages will be lost when it exits or crashes. To ensure that information is not lost, there are two things to note: We have to make queues and messages persistent.
Queue persistence, it is necessary to declare queues by argument durable=true when declaring queues to be persisted
Channel.queue_declare (queue= ' Hello ', durable=true)
Although this line of code itself is correct, it still does not run correctly. Because we've already defined a non-persistent queue called Hello. RABBITMQ does not allow you to redefine a queue with different parameters, it will return an error. But we now use a quick fix--with different names, such as Task_queue.
Channel.queue_declare (queue= ' task_queue ', durable=true)
We will then make changes to the corresponding declaration queue in the sending and receiving messages.
Message persistence, we need to set the Delivery_mode property to 2.
Channel.basic_publish (exchange= ',
routing_key= "Task_queue",
body=message, Properties=pika
. Basicproperties (
Delivery_mode = 2, # make message persistent
))
Note:
Setting a message to persistent does not completely guarantee that it will not be lost. The above code just tells RABBITMQ to save the message to the hard disk, but there is a very small interval between the time the RABBITMQ receives the message and the save. Because RABBITMQ not all messages use Fsync (2)-it is possible to simply save it to the cache and not necessarily write to the hard disk. does not guarantee true persistence, but it is enough to deal with our simple task force columns. If you must ensure persistence, you need to rewrite your code to support the transaction (transaction).
Fair DispatchBy default, if there are no special instructions, RABBITMQ just assigns the message to Woker, and RABBITMQ doesn't know if Woker can handle the message or how many of the worker's not responding. It blindly sent the first n-th message to the n-th consumer. You can use the Basic.qos method and set the Prefetch_count=1. This is to tell RABBITMQ that at the same time, do not send more than 1 messages to a worker (worker) until it has processed the previous message and responded. This way, RABBITMQ will distribute the message to the next idle worker (worker).
Channel.basic_qos (Prefetch_count=1)
Note:About the size of the work queue if all the worker handles the busy state, your queue will be filled up. You need to be aware of this problem, either by adding more workers (workers) or by using other policies.
SummaryOn the basis of the first chapter, we have added the following concepts to the work queue: In practice, queues can be processed in parallel, and we can enhance productivity persistence by increasing the number of worker: for security, we need to ensure that the queue is not lost, Then queues and messages must maintain persistent queue persistence: Specify Durable=true message persistence when declaring: Specify message properties, delivery_mode=2 message Acknowledgement: Maintain integrity, and notify RABBITMQ after worker processing completes This will keep the whole process intact and avoid risk. Worker processing, through the Basic_ack to confirm the message Fair Dispatch: RABBITMQ only responsible for distribution, and do not care whether the worker processing completed, so the worker needs to specify, I can handle at most at the same time how many tasks, how many tasks to deal with, Otherwise, RABBITMQ will be sent indefinitely, resulting in a crash. When the worker is processed, the Basic_qos (prefetch_count=1) is used to control the integrated code sending program
#!/usr/bin/env python #-*-coding:utf-8-*-# @Date: 2016-02-28 21:28:17 # @Author: mx (mx472756841@gmail.com) # @ link:http://www.shujutiyu.com.cn/# @Version: $Id $ import OS import pika import sys conn = None message = '. Joi
N (sys.argv[1:]) or "Hello world!" Try: # get Connection conn = Pika. Blockingconnection (Pika. Connectionparameters (' localhost ') # get Channel channel = Conn.channel () # Before sending the queue, you need to determine if the queue exists, and if no rabbitmq will be discarded, create the team first Column # @param: Durable queue persistence, default false Channel.queue_declare (' Task_queue ', durable=true) # Sending messages in RABBITMQ, not sending teams directly column, instead of sending the switch (Exchange), there is no more research here. ret = Channel.basic_publish (exchange= ', routing_key= ' task _queue ', Body=message, Properties=pika. Basicproperties (delivery_mode=2, # message persistence) Prin
T ' [x] Sent ' {0} ' '. Format (message) print RET except Exception, E: Raise e finally:if Conn:conn.close () Receiving programs
#!/usr/bin/env python #-*-coding:utf-8-*-# @Date: 2016-02-29 16:30:21 # @Author: mx (mx472756841@gmail.com) # @ link:http://www.shujutiyu.com/# @Version: $Id $ import OS import pika import Time conn = None def callback (CH, M Ethod, properties, body): "" "@ch: Channel channel, is sent out by the outside of the body to read the contents of the queue to do different operations" "" Prin T ' [x] recived%r% (body,) print ' [x] ch {0} '. Format (ch) print [x] method {0} '. Format (method) print [
X] Properties {0} ". Format (properties) Time.sleep (Body.count ('. ')) print ' [x] done%r '% (body,) ch.basic_ack (Delivery_tag=method.delivery_tag) Try: # get Connection conn = P Ika. Blockingconnection (Pika. Connectionparameters (' localhost ') # get Channel channel = Conn.channel () # Declare queue, repeat sound Ming will not complain, but there is no queue of direct access will be an error # @param: Durable queue persistence channel.queue_declare (' Task_queue ', durable=true) # set Channe L Process a message at the same time Channel.basic_qos (prefetch_count=1) # Get message # @no_ack, do not need to return confirmation messages, the default is False, set, there may be lost packet, not set, # when no confirmation message is returned, it is possible to duplicate processing or memory overflow. Channel.basic_consume (Callback, queue= ' Task_queue ') print ' [*] waiting for messages. To exit Press CTRL + C ' channel.start_consuming () except Exception, e:raise e finally:if conn:conn.cl
OSE ()
This chapter introduces here, in fact, there are a lot of available things, you can provide the following reference to view, you can also view the source code of Pika, to take the necessary.
4. Reference materialsChinese information: http://rabbitmq-into-chinese.readthedocs.org/zh_cn/latest/tutorials_with_python/[2]work_queues/official website information: http ://www.rabbitmq.com/tutorials/tutorial-two-python.html
