PHP rabbitMQ Tutorial-2

Work Queues (Work/task queue) (using php-amqplib)

In the first tutorial we wrote programs to send and receive messages from a named queue. In this one we'll createWork QueueThat will be used to distribute time-consuming tasks among multiple workers.

Last time, we wrote a program to send and receive messages from the specified queue. This time, we need to create a task queue to distribute time-consuming tasks to multiple "Workers.

The main idea behind Work Queues (aka:Task Queues) Is to avoid doing a resource-intensive task immediately and having to wait for it to complete. Instead we schedule the task to be done later. We encapsulateTaskAs a message and send it to a queue. A worker process running in the background will pop the tasks and eventually execute the job. When you run your workers the tasks will be shared between them.

The main idea of working Queue (also called task queue) is to avoidNowIt takes a long time to complete a resource-intensive task. Instead, we arrange such tasks.LaterComplete. We encapsulate tasks into messages and send them to a queue. A "worker" process running in the background extracts these tasks and completes them. When you run multiple workers, tasks are shared among them.

This concept is especially useful in web applications where it's impossible to handle a complex task during a short HTTP request window.

It is impossible to process a complex task in a short HTTP request. This concept is particularly useful in web applications.

Preparation (Preparation !)

In the previous part of this tutorial we sent a message containing "Hello World! ". Now we'll be sending strings that stand for complex tasks. we don't have a real-world task, like images to be resized or pdf files to be rendered, so let's fake it by just pretending we're re busy-by using the sleep () function. we'll take the number of dots in the string as its complexity; every dot will account for one second of "work ". for example, a fake task described by Hello... will take three seconds.

As mentioned above, we sent a "Hello World" message. This time, some strings are sent to represent complex tasks. We don't have tasks in real life like adjusting the image size or pdf rendering, so we use the sleep () function to pretend that the task is time-consuming and then simulate the real scenario.

We will slightly modifySend. phpCode from our previous example, to allow arbitrary messages to be sent from the command line. This program will schedule tasks to our work queue, so let's name it new_task.php:

Remember "send. php". Let's modify it a little so that it can send any message through commands. This program will schedule tasks to our work queue,

So call it new_task.php.

$data = implode(' ', array_slice($argv, 1));if(empty($data)) $data = "Hello World!";$msg = new AMQPMessage($data,                        array('delivery_mode' => 2) # make message persistent                      );$channel->basic_publish($msg, '', 'task_queue');echo " [x] Sent ", $data, "\n";

Our oldReceive. phpScript also requires some changes: it needs to fake a second of work for every dot in the message body. it will pop messages from the queue and perform the task, so let's call it worker. php:

The previous receive. php also needs to be adjusted: each vertex in the message needs to be treated as a second to simulate the time consumption. It extracts and executes tasks from the queue, so it is called worker. php.

$callback = function($msg){  echo " [x] Received ", $msg->body, "\n";  sleep(substr_count($msg->body, '.'));  echo " [x] Done", "\n";  $msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);};$channel->basic_qos(null, 1, null);$channel->basic_consume('task_queue', '', false, false, false, false, $callback);

Note that our fake task simulates execution time.

Note that we assume that the task needs to be simulated for execution time.

Run them as in tutorial one: the execution method is the same as the execution method (as shown below)

shell1$ php new_task.php "A very hard task which takes two seconds.."shell2$ php worker.php

Round-robin dispatching (Cyclic Scheduling/distribution)

One of the advantages of using a Task Queue is the ability to easily parallelise work. If we are building up a backlog of work, we can just add more workers and that way, scale easily.

The ability to work in parallel easily is one of the advantages of using task queues. If we have a bunch of tasks, we can add more workers, which is easy to measure.

First, let's try to run two worker. php scripts at the same time. They will both get messages from the queue, but how exactly? Let's see.

First, our colleagues run worker. php. All of them get messages from the queue. What is it? Next, let's take a look.

You need three dead les open. Two will run the worker. php script. These dead les will be our two consumers-C1 and C2.

Open three consoles. Run worker. php. The two consoles are our two consumers, namely C1 and C2.

shell1$ php worker.php [*] Waiting for messages. To exit press CTRL+C

shell2$ php worker.php [*] Waiting for messages. To exit press CTRL+C

In the third one we'll publish new tasks. Once you 've started the consumers you can publish a few messages:

Third, we use it to publish tasks. If you have already run the consumer, publish some messages.

shell3$ php new_task.php First message.shell3$ php new_task.php Second message..shell3$ php new_task.php Third message...shell3$ php new_task.php Fourth message....shell3$ php new_task.php Fifth message.....

Let's see what is delivered to our workers:

The worker sent to us.

shell1$ php worker.php [*] Waiting for messages. To exit press CTRL+C [x] Received 'First message.' [x] Received 'Third message...' [x] Received 'Fifth message.....'

shell2$ php worker.php [*] Waiting for messages. To exit press CTRL+C [x] Received 'Second message..' [x] Received 'Fourth message....'

By default, RabbitMQ will send each message to the next consumer, in sequence. on average every consumer will get the same number of messages. this way of distributing messages is called round-robin. try this out with three or more workers.

By default, RabbitMQ sends each message to the next consumer in order. Generally, each consumer receives the same number of messages. This message distribution method is called "round-robin ". Try three or more wokers.

Message acknowledgment (Message confirmation)

Doing a task can take a few seconds. you may wonder what happens if one of the consumers starts a long task and dies with it only partly done. with our current code, once RabbitMQ delivers a message to the customer it immediately removes it from memory. in this case, if you kill a worker we will lose the message it was just processing. we'll also lose all the messages that were dispatched to this particle worker but were not yet handled.

Processing a task takes several seconds. You may want a consumer to execute a time-consuming task and wait until it is finished. According to our current code,

Once RabbitMQ sends a message to a consumer, it will be deleted from the memory neutral. In this scenario, if you kill a worker process

The message being processed will be lost. Similarly, all messages sent to this worker will be lost before being processed.

But we don't want to lose any tasks. If a worker dies, we 'd like the task to be delivered to another worker.

We certainly don't want to lose any tasks. If a worker crashes, we hope that this task will be assigned to another worker.

In order to make sure a message is never lost, RabbitMQ supports messageAcknowledgments. An ack (nowledgement) is sent back from the consumer to tell RabbitMQ that a participant message has been received, processed and that RabbitMQ is free to delete it.

To ensure that messages are never lost, RabbitMQ supports message confirmation. What do you mean? That is to say, the consumer sends a message to RabbitMQ to tell it that a specific

The message has been received and processed, so that RabbitMQ can delete it at will.

If a consumer dies without sending an ack, RabbitMQ will understand that a message wasn' t processed fully and will redeliver it to another consumer. that way you can be sure that no message is lost, even if the workers occasionally die.

If the consumer fails to send (Message) confirmation, RabbitMQ considers that the message is not completely processed and resends it to another consumer. That way

Then you can ensure that no messages will be lost, even if the worker dies intermittently.

There aren't any message timeouts; RabbitMQ will redeliver the message only when the worker connection dies. It's fine even if processing a message takes a very, very long time.

When the worker connection fails, RabbitMQ resends the message. Therefore, message processing timeout does not exist. Even if it takes a long time

... Handle a message.

Message acknowledgments are turned off by default. It's time to turn them on by setting the fourth parameter to basic_consume to false (true meansNo ack) And send a proper acknowledgment from the worker, once we're done with a task.

By default, message validation is disabled. It is time to enable them. Once a task is completed, appropriate confirmation information is sent by setting the fourth parameter basic_consume to false (true means closing message confirmation.

$callback = function($msg){  echo " [x] Received ", $msg->body, "\n";  sleep(substr_count($msg->body, '.'));  echo " [x] Done", "\n";  $msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);};$channel->basic_consume('task_queue', '', false, false, false, false, $callback);

Using this code we can be sure that even if you kill a worker using CTRL + C while it was processing a message, nothing will be lost. soon after the worker dies all unacknowledged messages will be redelivered.

With this code, we can ensure that Shenma will not be lost, even if you use CTRL + C to kill the message when the worker processes it. If the worker is killed

All unconfirmed messages will be resdelivered.

Forgotten acknowledgment (elderly amnesia)

It's a common mistake to miss the basic_ack. it's an easy error, but the consequences are serious. messages will be redelivered when your client quits (which may look like random redelivery), but RabbitMQ will eat more and more memory as it won't be able to release any unacked messages.

It is common to lose basic_ack. It is a very common mistake, but the consequences are very serious !!!

When your client exits, the message will be re-delivered (it looks like a random resend), and RabbitMQ will eat more and more memory,

Because it will not release any unconfirmed messages !!!

In order to debug this kind of mistake you can use rabbitmqctl to print the messages_unacknowledged field:

To debug this error, you can use rabbitmqctl to print the messages_unacknowledged field.

$ sudo rabbitmqctl list_queues name messages_ready messages_unacknowledgedListing queues ...hello    0       0...done.

Message durability)

We have learned how to make sure that even if the consumer dies, the task isn't lost. But our tasks will still be lost if RabbitMQ server stops.

We have learned how to ensure that tasks are not lost when the consumer fails. However, if the RabbitMQ service fails, our tasks will still be lost.

When RabbitMQ quits or crashes it will forget the queues and messages unless you tell it not. two things are required to make sure that messages aren't lost: we need to mark both the queue and messages as durable.

When RabbitMQ exits or goes down, it will forget the queue and message, unless you tell it !!!

To ensure that messages are not lost, we need to do two things: we need to mark both queues and messages as persistent.

First, we need to make sure that RabbitMQ will never lose our queue. In order to do so, we need to declare itDurable. To do so we pass the third parameter to queue_declare as true:
First, we need to ensure that RabbitMQ will never lose our queue. To do this, we have to declare it persistent. That is, queue_declare

The third parameter is set to true.

$channel->queue_declare('hello', false, true, false, false);

Although this command is correct by itself, it won't work in our present setup. that's because we 've already defined a queue called hello which is not durable. rabbitMQ doesn't allow you to redefine an existing queue with different parameters and will return an error to any program that tries to do that. but there is a quick workaround-let's declare a queue with different name, for example task_queue:

Although this is correct, the current setting does not work. Because we have defined a non-persistent queue named hello. RabbitMQ does not allow you to use different parameters to redefine an existing queue and return an error to any program trying to do so. However, there is a quick solution-we can declare a queue with different names, such as task_queue:

$channel->queue_declare('task_queue', false, true, false, false);

This flag set to true needs to be applied to both the producer and consumer code.

Both the producer and consumer code must set the queue declaration here to true (the third parameter ).

At this point we're re sure that the task_queue queue won't be lost even if RabbitMQ restarts. now we need to mark our messages as persistent-by setting the delivery_mode = 2 message property which AMQPMessage takes as part of the property array.

At this point, we can ensure that the queue will not be lost when RabbitMQ restarts task_queue in a timely manner. Now, we have to mark the message as persistent, and set the attribute delivery_mode = 2 of an array as the parameter of AMQPMessage (second.

$msg = new AMQPMessage($data,       array('delivery_mode' => 2) # make message persistent       );

Note on message persistence (delete)

Marking messages as persistent doesn't fully guarantee that a message won't be lost. although it tells RabbitMQ to save the message to disk, there is still a short time window when RabbitMQ has accepted a message and hasn't saved it yet. also, RabbitMQ doesn' t do fsync (2) for every message -- it may be just saved to cache and not really written to the disk. the persistence guarantees aren't strong, but it's more than enough for our simple task queue. if you need a stronger guarantee you can wrap the publishing code inTransaction.

Setting a message as persistent does not completely guarantee that it will not be lost. As soon as possible, it also tells RabbitMQ to save the message to the hard disk, but there are still

A very small possibility is that RabbitMQ has received the message but has not been able to save it to the hard disk, rabbitMQ does not perform fsync for each message-it is likely that it is stored in the cache rather than written to the hard disk. This persistence guarantee is not solid enough, but for us

A simple task queue is more than enough. If you need a more solid persistence guarantee, you can wrap the sending code in a transaction.

Fair dispatch (Fair allocation)

You might have noticed that the dispatching still doesn' t work exactly as we want. for example in a situation with two workers, when all odd messages are heavy and even messages are light, one worker will be constantly busy and the other one will do hardly any work. well, RabbitMQ doesn' t know anything about that and will still dispatch messages evenly.

You may find that this distribution task is not running as expected. For example, if there are two workers, when all the odd messages are very time-consuming, even if it is not time-consuming, one worker will remain busy and the other will barely work. Haomiao, RabbitMQ has no idea about this situation and will continue sending even messages.

This happens because RabbitMQ just dispatches a message when the message enters the queue. it doesn't look at the number of unacknowledged messages for a consumer. it just blindly dispatches every n-th message to the n-th consumer.

The reason for this is that when the information enters the queue RabbitMQ, it is only for distribution. It does not view unconfirmed messages of a consumer. It will only blindly distribute the nth message to the nth consumer.

In order to defeat that we can use the basic_qos method with the prefetch_count = 1 setting. this tells RabbitMQ not to give more than one message to a worker at a time. or, in other words, don't dispatch a new message to a worker until it has processed and acknowledged the previous one. instead, it will dispatch it to the next worker that is not still busy.

To solve this problem, we can use the basic_qus method to set prefetch_count = 1. This will tell RabbitMQ to send only one worker message at a time.

Or, in other words, do not send a new message to a worker before it finishes processing and confirms the previous message. Instead, RabbitMQ delivers messages to the next

Idle worker.

$channel->basic_qos(null, 1, null);

Note about queue size (Note it !! Queue size)

If all the workers are busy, your queue can fill up. You will want to keep an eye on that, and maybe add more workers, or have some other strategy.

If all workers are busy, your queue will be filled. Pay attention to either adding more workers or some other strategies.

Putting it all together (fit !!! Again haha)

Final code of our new_task.php file :( new_task.php's ultimate God code)

 channel();$channel->queue_declare('task_queue', false, true, false, false);$data = implode(' ', array_slice($argv, 1));if(empty($data)) $data = "Hello World!";$msg = new AMQPMessage($data,                        array('delivery_mode' => 2) # make message persistent                      );$channel->basic_publish($msg, '', 'task_queue');echo " [x] Sent ", $data, "\n";$channel->close();$connection->close();?>

(New_task.php source)

And our worker. php: (worker)

 channel();$channel->queue_declare('task_queue', false, true, false, false);echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";$callback = function($msg){  echo " [x] Received ", $msg->body, "\n";  sleep(substr_count($msg->body, '.'));  echo " [x] Done", "\n";  $msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);};$channel->basic_qos(null, 1, null);$channel->basic_consume('task_queue', '', false, false, false, false, $callback);while(count($channel->callbacks)) {    $channel->wait();}$channel->close();$connection->close();?>

(Worker. php source)

Using message acknowledgments and prefetch you can set up a work queue. The durability options let the tasks keep ve even if RabbitMQ is restarted.

You can use message confirmation and pre-capture to create a work queue. The persistence settings make the task alive even if RabbitMQ restarts.

Now we can move on to tutorial 3 and learn how to deliver the same message to consume consumers.

Next, we will learn how to send the same message to multiple consumers, just like broadcast, well!