Translation-php RabbitMQ tutorial-2_php tutorial

Work Queues (Job/task queue)

(using Php-amqplib)

In the first tutorial we wrote programs to send and receive messages from a named queue. In this one we'll create a work Queue that'll be used to distribute time-consuming tasks among multiple workers .

Last time, we wrote a program to send and receive messages from a specified queue. This time, we're going to create a task queue that can be used to distribute the task to multiple "workers".

The main idea behind work Queues (AKA: task Queues) was to avoid doing a resource-intensive Task immediately and H Aving to wait for it to complete. Instead We schedule the task to is done later. We encapsulate a task as a message and send it to a queue. A worker process running in the background would pops the tasks and eventually execute the job. When you run many workers the tasks would be shared between them.

The main idea of a work queue (also called a task queue) is to avoid immediate processing of tasks that are resource-intensive and require a long processing time to complete. On the contrary, we arrange for such a task to be completed later . We send the task into a queue in the form of message encapsulation. A "worker" process running in the background takes out these tasks and eventually completes the task. When you run more than one worker, the tasks are shared between them.

This concept was especially useful in Web applications where it's impossible to handle a complex task during a short HTTP R Equest window.

It is not possible to handle a complex task in a shorter HTTP request, which is especially useful in Web applications.

Preparation (Ready!) ）

In the previous section of this tutorial we sent a message containing "Hello world!". Now we'll be sending strings this stand for complex tasks. We don ' t have a real-world task, like images to being resized or PDF files to being rendered, so let's fake it by just Pretendin G we ' re busy-by using the Sleep () function. We ll take the number of dots in the string as its complexity; Every dot would account for one second of the "work". For example, a fake task described by Hello ... would take three seconds.

Last said, we sent a "Hello world" news. This sends some strings to represent complex tasks. We don't have real-life tasks like resizing images or pdf rendering, so it's time-consuming to disguise tasks with the sleep () function to simulate real-world scenarios.

We'll slightly modify the send.php code from our previous example, to allow arbitrary messages to being sent from The command line. This program would schedule tasks to our work queue and so let's name it new_task.php:

Remember "send.php", let's make a little change down so that it can send any message by command. This program will schedule tasks in our work queue,

So just call it new_task.php.

$data = Implode (' ', Array_slice ($ARGV, 1)), if (Empty ($data)) $data = "Hello world!"; $msg = new Amqpmessage ($data,                        array (' delivery_mode ' = 2) # make message persistent                      ); $channel->basic_ Publish ($msg, ' ', ' task_queue '); echo "[X] Sent", $data, "\ n";

Our old receive.php script also requires some changes:it needs to fake a second of work for every dot in the MES Sage Body. It would pop messages from the queue and perform the task, so let's call it worker.php:

The previous receive.php also made some adjustments: each dot in the message needs to be treated as a second to simulate the time-consuming process. It pulls out of the queue and executes the task, so call it worker.php.

$callback = function ($msg) {  echo "[x] Received", $msg->body, "\ n";  Sleep (Substr_count ($msg->body, '. '));  echo "[x] Done", "\ n";  $msg->delivery_info[' channel ']->basic_ack ($msg->delivery_info[' Delivery_tag ');}; $channel->basic_qos (NULL, 1, NULL), $channel->basic_consume (' Task_queue ', ' ", False, False, False, False, $ callback);

Note that our fake task simulates execution time.

Note that our hypothetical task needs to simulate execution time.

Run them as in tutorial one: performs the same way as last (see below)

shell1$ php new_task.php "A very hard task which takes the seconds ..." shell2$ PHP worker.php

Round-robin dispatching (cyclic dispatch/distribution)

One of the advantages of using a Task Queue is the ability to easily parallelise work. If We are building to a backlog of work, we can just the add more workers and the that, the scale easily.

Working in parallel with ease is one of the advantages of using the task queue. If we had a bunch of tasks, it would be nice to add more workers and it would be easy to measure.

First, let's try to run a worker.php scripts at the same time. They would both get messages from the queue, but how exactly? Let ' s see.

First, our colleagues run two worker.php. They all get the message from the queue, but what is meow? Then look.

You need three consoles open. The worker.php script is run by the. These consoles'll is our consumers-c1 and C2.

Open three consoles. Two of them run worker.php. These two consoles are our two consumers, called C1 and C2 bar.

shell1$ php worker.php [*] waiting for messages. To exit Press CTRL + C

shell2$ php worker.php [*] waiting for messages. To exit Press CTRL + C

In the third one we ll publish new tasks. Once you ' ve started the consumers you can publish a few messages:

The third one, we use to publish the task. If you have already run the consumer, just post some news.

shell3$ php new_task.php First message.shell3$ php new_task.php Second message: shell3$ php new_task.php Third message...shell3$ php new_task.php fourth message....shell3$ php new_task.php Fifth message .....

Let's see what's delivered to our workers:

Look at what's been sent to our worker.

shell1$ php worker.php [*] waiting for messages. To exit Press CTRL + C [x] Received ' first message. ' [x] Received ' third message ... ' [x] Received ' fifth message ... '

shell2$ php worker.php [*] waiting for messages. To exit Press CTRL + C [x] Received ' Second message: ' [x] Received ' fourth message .... '

By default, RabbitMQ would send each message to the next consumer, in sequence. On average every consumer would get the same number of messages. This is the distributing messages is called Round-robin. Try this out with three or more workers.

By default, RabbitMQ sends each message sequentially to the next consumer, usually receiving the same number of messages per consumer. This way of distributing messages is called "Round-robin". Try three or more woker.

Message acknowledgment (confirmation of messages)

Doing a task can take a few seconds. Wonder what happens if one of the consumers starts a long task and dies with it is only partly done. With our current code, once RabbitMQ delivers a message to the customer it immediately removes it from memory. If you kill a worker we'll lose the message it was just processing. We'll also lose all the messages so were dispatched to this particular worker but were not yet handled.

It takes several seconds to process a task. You might want to be one of the consumers who performs a time-consuming task and dies halfway through the whole. As far as our code is concerned,

Once RABBITMQ sends a message to a consumer, it is removed from memory immediately. In this scenario, if you k-drop a worker process, then it

The message being processed will be lost. Also, all messages sent to this worker will be lost before they are processed.

But we don ' t want to lose any tasks. If A worker dies, we ' d like the task to being delivered to another worker.

We certainly don't want to lose any task. If a worker is hung up, we would very much like this task to be assigned to other worker heads.

In order to make sure a message is never lost, RabbitMQ supports message acknowledgments. An ACK (nowledgement) was sent back from the consumer to tell RabbitMQ that a particular message had been received, Processe D and that RabbitMQ are free to delete it.

To ensure that messages are never lost, RABBITMQ supports message acknowledgement. What do you mean? That means the consumer sends a message to RABBITMQ to tell it that a particular

The message has been received and processed so that RABBITMQ can easily delete it.

If a consumer dies without sending an ACK, RabbitMQ would understand that a message wasn ' t processed fully and would redeliv Er it to another consumer. That's the sure that no message is lost, even if the workers occasionally die.

If the consumer is unable to send a confirmation, RABBITMQ will assume that the message was not fully processed, and then re-dispatch it to another consumer. That

Then you can promise not to lose the message, even if the worker dies intermittently.

There aren ' t any message timeouts; RabbitMQ would redeliver the message only when the worker connection dies. It ' s fine even if processing a message takes a very, very long time.

RABBITMQ will resend the message only when the worker connection is suspended. Therefore, there is no message processing timeout issue. Even if it takes a long, long time.

.... Handling a message is all about wood.

Message acknowledgments is turned off by default. It ' s time to turn them on by setting the fourth parameter to Basic_consume to False (true means no ACK) and send A proper acknowledgment from the worker, once we ' re do with a task.

By default, message acknowledgement is turned off. It's time to open them up. Once a task is completed, set the Basic_consume fourth parameter to False (true to turn off message acknowledgement) to send the appropriate acknowledgement.

$callback = function ($msg) {  echo "[x] Received", $msg->body, "\ n";  Sleep (Substr_count ($msg->body, '. '));  echo "[x] Done", "\ n";  $msg->delivery_info[' channel ']->basic_ack ($msg->delivery_info[' Delivery_tag ');}; $channel->basic_consume (' task_queue ', ', False, False, False, false, $callback);

Using this code we can is sure that even if you kill a worker using CTRL + C while it is processing a message, nothing would Be lost. Soon after the worker dies all unacknowledged messages would be redelivered.

With this code we can guarantee that God's horse will never be lost, even if you drop it with CTRL + C while the worker is handling the message. If the worker is killed, soon

All unconfirmed messages will be re-dispatched.

Forgotten acknowledgment (old age amnesia)

It ' s a common mistake to miss the Basic_ack. It's an easy error, but the consequences is serious. Messages would be redelivered when your client quits (which could look like random redelivery), but RabbitMQ would eat more an d more memory as it won ' t is able to release any unacked messages.

Missing Basic_ack is common. is a very tolerant mistake, but the consequences are very serious!!!

When your client exits, the message will be re-dispatched (it looks like a random re-Dispatch), and RABBITMQ will eat more and more of your memory.

Because it will not release any unconfirmed messages WOW!!!

In order to debug this kind of mistake your can use Rabbitmqctl to print the messages_unacknowledged field:

To debug this error, you can use Rabbitmqctl to print the Messages_unacknowledged field.

$ sudo rabbitmqctl list_queues name messages_ready messages_unacknowledgedlisting queues ... hello    0       0...done.

Message durability (Messaging persistence)

We have learned how to make sure that even if the consumer dies, the task isn ' t lost. But our tasks would still be lost if RabbitMQ server stops.

We've learned how to make sure that the task is not lost when the consumer hangs out. But if the RABBITMQ service hangs, our task will still be lost.

When RabbitMQ quits or crashes it would forget the queues and messages unless you tell it not to. The things is required to make sure this messages aren ' t lost:we need to mark both the queue and messages as durable.

When RABBITMQ quits or hangs out, it forgets the queue and messages, unless you tell it the table Jiangzi!!!

To ensure that messages are not lost, you need to do two things: we need to mark both the queue and the message as persistent.

First, we need to make sure that RabbitMQ would never lose our queue. In order to does so, we need to declare it as durable. To-do and we pass the third parameter to Queue_declare as true:
First, we need to make sure that rabbitmq never loses our queue. To do so, we have to declare it to be durable. Which is to put Queue_declare

The third parameter is set to True.

$channel->queue_declare (' Hello ', false, True, false, false);

Although this command was correct by itself, it won ' t work in our present setup. That's because we ve already defined a queue called Hello which is not durable. RabbitMQ doesn ' t allow you to redefine a existing queue with different parameters and would return an error to any program That's tries to does that. But there was a quick Workaround-let ' s declare a queue with different name, for example Task_queue:

Although this is true, the current settings do not work. Because we have defined a non-persistent queue called Hello. RABBITMQ does not allow a different parameter to redefine an existing queue and returns an error to any program that attempts to do so. But there is a quick solution-we can declare a queue of different names, such as Task_queue:

$channel->queue_declare (' Task_queue ', False, True, false, false);

This flag set is true needs to being applied to both the producer and consumer code.

Both the producer and consumer code need to set the queue declaration here to true (the third parameter).

At this point we ' re sure that the Task_queue queue won ' t be lost even if RabbitMQ restarts. Now we need to mark our messages as persistent-by setting the Delivery_mode = 2 message property which Amqpmessage takes As part of the property array.

In this case, we can guarantee that the RABBITMQ will be restarted in time Task_queue this queue is not lost. Now, we have to mark the message as persistent, set the property of an array delivery_mode=2, as the Amqpmessage parameter (the second one).

$msg = new Amqpmessage ($data,       array (' delivery_mode ' = 2) # make message persistent       );

Note on message persistence (requires attention)

Marking messages as persistent doesn ' t fully guarantee that a message won ' t is lost. Although it tells RabbitMQ to save the message to disk, there was still a short time window time RabbitMQ have accepted a me Ssage and hasn ' t saved it yet. Also, RabbitMQ doesn ' t do fsync (2) for every message--it could be just saved to cache and not really written to the disk. The persistence guarantees aren ' t strong, but it's more than enough for our simple task queue. If you need a stronger guarantee your can wrap the publishing code in a transaction.

Setting the message to persistent does not guarantee that it will not be lost. As soon as it also told RABBITMQ to save the message to the hard drive, but there is still

A small possibility exists-that is, RABBITMQ receives the message but has not yet been able to save it to the hard disk, and RABBITMQ does not do fsync for every message-it is likely to just be saved in the cache instead of actually writing to the hard disk. This guarantee of durability is not strong enough, but for us

The simple task queue is more than sufficient. If you need more solid durability guarantees, you can wrap the sending code in one transaction.

Fair Dispatch (fair distribution)

You might has noticed that the dispatching still doesn ' t work exactly as we want. For example in a situation with both workers, when all odd messages is heavy and even messages is light, one worker would Be constantly busy and the other one would do hardly any work. Well, RabbitMQ doesn ' t know anything on that and would still dispatch messages evenly.

You may have found out that the distribution was not running as I expected. For example, a situation with two workers, when all the odd messages are very time-consuming, even if it is not time-consuming, one worker will be busy, and the other almost do not work. Good meow, rabbitmq. No knowledge of this situation will continue to distribute the message evenly.

This happens because RabbitMQ just dispatches a message when the message enters the queue. It doesn ' t look at the number of unacknowledged messages for a consumer. It just blindly dispatches every n-th message to the n-th consumer.

This happens because the information goes into the queue rabbitmq just for distribution. It doesn't look at a consumer unconfirmed message. It will only blindly distribute nth messages to nth consumers.

In order to defeat, we can use the Basic_qos method with the Prefetch_count = 1 setting. This tells RabbitMQ not to give more than one message to a worker at a time. Or, in other words, don ' t dispatch a new message to a worker until it has processed and acknowledged the previous one. Instead, it'll dispatch it to the next worker that's not still busy.

To solve this situation, we can use the Basic_qus method to set the Prefetch_count=1. This tells RABBITMQ to give only one worker a message at a time.

Or, in other words, do not send a new message to a worker until it finishes processing and confirming the previous message. Instead, RABBITMQ will send the message to the next

The idle worker.

$channel->basic_qos (NULL, 1, NULL);

Note about queue size (attention!!) Queue size)

If all the workers is busy, your queue can fill up. You'll want to keep a eye on this, and maybe add more workers, or has some other strategy.

If all the workers are busy, your queue will be filled. Keep an eye on it, either by adding more workers or some other strategy.

Putting it all together (fitness!!!) again haha)

Final code of our new_task.php file: (new_task.php's Ultimate Divine Code)

 
  Channel (); $channel->queue_declare (' Task_queue ', False, True, false, false); $data = Implode (", Array_slice ($ARGV, 1 ); if (empty ($data)) $data = "Hello world!"; $msg = new Amqpmessage ($data,                        array (' delivery_mode ' = 2) # make message persistent                      ); $channel->basic_ Publish ($msg, ' ', ' task_queue '); echo "[X] Sent", $data, "\ n"; $channel->close (); $connection->close (); >

(new_task.php Source)

And our worker.php: (Worker Oh)

 
  Channel (); $channel->queue_declare (' Task_queue ', False, True, false, false); Echo ' [*] waiting for messages. To exit Press CTRL + C ', "\ n"; $callback = function ($msg) {  echo "[x] Received", $msg->body, "\ n";  Sleep (Substr_count ($msg->body, '. '));  echo "[x] Done", "\ n";  $msg->delivery_info[' channel ']->basic_ack ($msg->delivery_info[' Delivery_tag ');}; $channel->basic_qos (NULL, 1, NULL), $channel->basic_consume (' Task_queue ', ' ", False, False, False, False, $ callback), while (count ($channel->callbacks)) {    $channel->wait ();} $channel->close (); $connection->close ();? >

(worker.php Source)

Using message acknowledgments and prefetch you can set up a work queue. The durability options let the tasks survive even if RabbitMQ is restarted.

You can use message confirmations and pre-fetching to create a work queue. Persistent settings allow the task to survive even if the RABBITMQ is restarted.

Now we can move on to Tutorial 3 and learn how to deliver the same message to many consumers.

Next, we'll learn how to send the same message to multiple consumers, just like the broadcast, uh-huh!

http://www.bkjia.com/PHPjc/755776.html www.bkjia.com true http://www.bkjia.com/PHPjc/755776.html techarticle work Queues (working/Task queue) (using Php-amqplib) in the first tutorial we wrote programs to send and receive messages from a Named queue. in this one we'll create a work Queue ...

