Rabbitmq method of Use (ii)

Work Queues

In the first tutorial we wrote programs to send and receive messages from a named queue. In this one we'll create a work Queue that'll be used to distribute time-consuming tasks among multiple wor Kers.

The main idea behind work Queues (AKA: task Queues) was to avoid doing a resource-intensive Task immediately a ND has to-wait for it to complete. Instead We schedule the task to is done later. We encapsulate a task as a message and send it to the queue. A worker process running in the background would pops the tasks and eventually execute the job. When you run many workers the tasks would be shared between them.

Round-robin dispatching

One of the advantages of using a Task Queue is the ability to easily parallelise work. If We are building to a backlog of work, we can just the add more workers and the that, the scale easily.

by default, RabbitMQ would send each message to the next consumer, in sequence. On average every consumer would get the same number of messages. This is the distributing messages is called Round-robin.

Message Acknowledgment

Doing a task can take a few seconds. Wonder what happens if one of the consumers starts a long task and dies with it is only partly done. With our current code once RabbitMQ delivers message to the customer it immediately removes it from memory. If you kill a worker we'll lose the message it was just processing. We'll also lose all the messages so were dispatched to this particular worker but were not yet handled.

But we don ' t want to lose any tasks. If A worker dies, we ' d like the task to being delivered to another worker.

In order to make sure a message is never lost, RabbitMQ supports message acknowledgments. An ACK (nowledgement) was sent back from the consumer to tell RabbitMQ that a particular message had been received, Processe D and that RabbitMQ are free to delete it.

If consumer dies without sending an ACK, RabbitMQ would understand that a message wasn ' t processed fully and would redeliver it to another consumer. That's the sure that no message is lost, even if the workers occasionally die.

There aren ' t any message timeouts; RabbitMQ would redeliver the message only when the worker connection dies. It ' s fine even if processing a message takes a very, very long time.

Message Durability

We have learned how to make sure that even if the consumer dies, the task isn ' t lost. But our tasks would still be lost if RabbitMQ server stops.

When RabbitMQ quits or crashes it would forget the queues and messages unless you tell it not to. The things is required to make sure this messages aren ' t lost:we need to mark both the queue and messages as durable.

First, we need to make sure that RabbitMQ would never lose our queue. In order to does so, we need to declare it as durable:

1 channel.queue_declare (queue='hello', durable=true)

Although this command was correct by itself, it won ' t work in our setup. That's because we ve already defined a queue called Hello which is not durable. RabbitMQ doesn ' t allow you to redefine a existing queue with different parameters and would return an error to any program that's tries to does that. But there was a quick Workaround-let ' s declare a queue with different name

This queue_declare change needs to being applied to both the producer and consumer code.

At the point we ' re sure this the task_queue queue won ' t be lost even if RabbitMQ restart S. Now we need to mark our messages as persistent – by supplying A delivery_mode  property with a value 2.

1 channel.basic_publish (exchange=",2                       routing_key="task_ Queue",3                       body=message,4                       properties=pika. Basicproperties (5                          # make message persistent6                       ))

Fair Dispatch

You might has noticed that the dispatching still doesn ' t work exactly as we want. For example in a situation with both workers, when all odd messages is heavy and even messages is light, one worker would Be constantly busy and the other one would do hardly any work. Well, RabbitMQ doesn ' t know anything on that and would still dispatch messages evenly.

This happens because RabbitMQ just dispatches a message when the message enters the queue. It doesn ' t look at the number of unacknowledged messages for a consumer. It just blindly dispatches every n-th message to the n-th consumer.

In order to defeat, we can use the Basic.qos method with the prefetch_count=1 setting. This tells RabbitMQ not to give more than one message to a worker at a time. Or, in other words, don ' t dispatch a new message to a worker until it has processed and acknowledged the previous ONE. Instead, it'll dispatch it to the next worker that's not still busy.

1 channel.basic_qos (prefetch_count=1)

Putting it all together

Final Code of our new_task.py script:

1 #!/usr/bin/env python2 ImportPika3 ImportSYS4 5Connection =Pika. Blockingconnection (Pika. Connectionparameters (6host='localhost'))7Channel =Connection.channel ()8 9Channel.queue_declare (queue='Task_queue', durable=True)Ten  OneMessage =' '. Join (sys.argv[1:])or "Hello world!" AChannel.basic_publish (exchange="', -routing_key='Task_queue', -body=message, theproperties=Pika. Basicproperties ( -Delivery_mode = 2,#Make message Persistent -                       )) - Print "[x] Sent%r"%(message,) +Connection.close ()

And our worker: (new_task.py Source)

1 #!/usr/bin/env python2 ImportPika3 Import Time4 5Connection =Pika. Blockingconnection (Pika. Connectionparameters (6host='localhost'))7Channel =Connection.channel ()8 9Channel.queue_declare (queue='Task_queue', durable=True)Ten Print '[*] waiting for messages. To exit Press CTRL + C' One  A defCallback (ch, method, properties, body): -     Print "[x] Received%r"%(body,) -Time.sleep (Body.count ('.') ) the     Print "[x] done" -Ch.basic_ack (Delivery_tag =Method.delivery_tag) -  -Channel.basic_qos (prefetch_count=1) + Channel.basic_consume (Callback, -Queue='Task_queue') +  AChannel.start_consuming ()

Rabbitmq method of Use (ii)