RabbitMQ Original article 03 -- work queue, rabbitmq03 --

RabbitMQ Original article 03 -- work queue, rabbitmq03 --

Work queue:

In the previous article, we created a program to send and receive messages in the named queue. In this article, I will create a work queue to allocate time-consuming operations to multiple executors.

The main idea of working Queue (task queue) is to avoid immediately executing resource-intensive tasks and have to wait for them to complete. Instead, they should be arranged for delayed execution. We pack (compress) these tasks into messages and send them to the Message Queue. Finally, the worker processes running on the backend can get the messages and process them. Multiple worker processes can share this task.

This concept is very useful for web applications, because it is basically impossible to complete a very complex and time-consuming task in a very short http web request process.

Preparations

In the previous article, we sent a message containing "hello word". Now we will send a string representing a complex task. We have no real-world task scenarios, we want to modify the image size and output the pdf file, so let's use Thread. sleep assumes that we handle a lot of tasks. Let's use ". "number to indicate the magnitude of the task, ". "indicates that the task needs to be executed for 1 second, such as" Hello... "indicates that this task takes 3 seconds to complete.

We can slightly modify the previous sending code so that he can send any message from the command line. This program will schedule the task to our work queue, therefore, we modify the program file name NewTask. cs.

var message = GetMessage(args);var body = Encoding.UTF8.GetBytes(message);var properties = channel.CreateBasicProperties();properties.SetPersistent(true);channel.BasicPublish(exchange: "",routingKey: "task_queue",basicProperties: properties,body: body);

Then, to help get messages from command line parameters

private static string GetMessage(string[] args){    return ((args.Length > 0) ? string.Join(" ", args) : "Hello World!");}

Then ourReceive. csYou also need to make some changes. when parsing a message to a ".", you need to pretend to execute 1 second of work. It will process messages sent from RabbitMQ, so we call it Worker. cs

var consumer = new EventingBasicConsumer(channel);consumer.Received += (model, ea) =>{    var body = ea.Body;    var message = Encoding.UTF8.GetString(body);    Console.WriteLine(" [x] Received {0}", message);    int dots = message.Split('.').Length - 1;    Thread.Sleep(dots * 1000);    Console.WriteLine(" [x] Done");};channel.BasicConsume(queue: "task_queue", noAck: true, consumer: consumer);

We simulate the execution time for a fake task.

int dots = message.Split('.').Length - 1;Thread.Sleep(dots * 1000);

Note: The queue name "task_queue" is a new queue to distinguish it from the previous code.

Cyclic allocation

One advantage of using task queue is that we can execute tasks in parallel. If we generate a large number of tasks waiting for processing, we can directly join more workers to process the tasks, it is very easy to expand.

First, let's run two Work. cs programs at the same time. They will all be able to get tasks from the queue. How does that Work? Let's see:

You need to open three console programs, two of which run the Worker. cs program, which is the consumer of our message.

The third is our message producer.

Then we can use our producer to send some messages:

Then observe the output of our two Worker. cs programs:

By default, RabbitMQ sends messages to the next consumer in an ordered manner, so each consumer receives a consistent number of messages on average. This message sending mode is called "Round Robin ".

Message confirmation

A job may last for several seconds. You may wonder what will happen if a long-running job suddenly stops running (only partially executed, in our current Code, once RabbitMQ sends the message to the consumer, it will immediately delete it from the memory. In this case, once you terminate a working program, then we will lose the messages it is processing, and we will also lose all the messages allocated to this executor but not completed yet.

However, we do not want to lose the rest. If an executor terminates, we want the message to be passed to the next executor.

To ensure that messages are never lost, RabbitMQ supports message confirmation (AcknowledgmentsAfter a message is confirmed to be accepted and executed, a consumer can send an ack (nowledgement) to notify RabbitMQ that the message has been completed and RabbitMQ can delete it.

If a consumer terminates (the session is closed, the connection is closed, or the tcp connection is lost) and ack is not sent, RabbitMQ will know that the message has not been completely processed, it will re-queue the message. If another consumer is online at the same time, it will immediately send it to another consumer. In this way, you can ensure that no message is lost when the consumer terminates unexpectedly.

There is no message timeout, and RabbitMQ will push the message again when a consumer terminates. There is no problem when a bill is processed for a long time.

Automatic Message confirmation is enabled by default. in the previous case, We disabled this option and set noack to true. Now it is time to change this parameter so that the executor can send acknowledgment to RabbitMQ.

var consumer = new EventingBasicConsumer(channel);                consumer.Received += (model, ea) =>                {                    var body = ea.Body;                    var message = Encoding.UTF8.GetString(body);                    Console.WriteLine("Thread:{0} [x] Received {1}", Thread.CurrentThread.GetHashCode(), message);                    int dots = message.Split('.').Length - 1;                    Thread.Sleep(dots * 1000);                    Console.WriteLine("Thread:{0} [x] Done",Thread.CurrentThread.GetHashCode());                    channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);                };                channel.BasicConsume(queue: "task_queue",noAck: false,consumer: consumer);

With this method, we can ensure that when we close the consumers who are executing the task in time, we can also ensure that there is no message to view, and all unacknowledged status messages will be redistributed by RabbitMQ.

Forgot to send confirmation: Forgetting to execute BasicAck is a very common problem, but its impact is very serious, when the client exits, the message will be re-distributed (it looks like a random allocation), but because RabbitMQ cannot release unacked messages, RabbitMQ will continue to eat more and more memory.

Message persistence

We have learned how to ensure that messages are not lost when the consumer ends, but our messages will still be lost when our RabbitMQ server stops.

When the RabbitMQ server exits or crashes, it will lose the messages and queues it stores, unless you specify it not to do so. To ensure that messages are not lost, we need to do two things to mark the queue and message persistence (durable ).

First, to ensure that the queue is not lost, we need to declare a persistent queue.

channel.QueueDeclare(queue: "task_queue", durable: true, exclusive: false, autoDelete: false, arguments: null);

Although this command is correct, it won't be executed successfully in our current environment, this is because we have declared a non-persistent queue named "hello. When any program tries to declare an existing queue using different parameters, RabbitMQ will return an error message.

So we can declare a queue with different names.

channel.QueueDeclare(queue: "task_queue", durable: true, exclusive: false, autoDelete: false, arguments: null);

Both the producer and consumer of the message need to apply the QueueDeclare command.

Now we confirm that even if the RabbitMQ server is accidentally terminated, we will not lose our "task_queue" queue. Now we need to set our message persistence-by setting IBasicProperties. setPersistent = true.

var properties = channel.CreateBasicProperties();properties.SetPersistent(true);

Message Persistence:

The message persistence tag does not guarantee that the message will not be lost. Although it tells RabbitMQ to make the message persistent, it still exists for a moment-RabbitMQ has accepted the message but has not saved it yet, similarly, RabbitMQ does not perform fsync (synchronize memory data to the storage device) for every incoming message in time. It may only store the message in the memory rather than on the disk. Although persistence is not robust enough, it is sufficient for simple programs. If you want more advanced message persistence confirmation, you can use publisher confirms.

Fair Scheduling

You may have noticed that this distribution mode is still not very consistent with what we want to achieve. For example, in a scenario, all the odd messages are heavyweight (long execution time ), all even-numbered tasks are short-lived messages, so one of them will be continuously in the busy State, while the other message processor can hardly perform complex tasks.

The reason for this is that RabbitMQ only sends messages in the message queue to the consumer, Instead of viewing the number of unacknowledged messages of the corresponding consumer, it only blindly sends the n-th message to the n-th consumer.

To avoid this problem, we can use the basicQos method and set its parameter prefetchCount = 1. This will tell RabbitMQ not to push more than one message to a consumer at the same time, in other words, when a processor is processing a message and has not sent acknowledged to RabbitMQ to confirm that the message has been completed, do not send another message to it. Correspondingly, RabbitMQ sends messages to the next idle consumer.

channel.BasicQos(0, 1, false);

Note:

When all workers are busy, the size of the queue continues to grow. Therefore, observe the queue status to determine whether to add new consumers or adopt other corresponding policies.

Complete code:

NewTask. cs:

Using System; using RabbitMQ. client; using System. text; class NewTask {public static void Main (string [] args) {var factory = new ConnectionFactory () {HostName = "localhost"}; using (var connection = factory. createConnection () using (var channel = connection. createModel () {channel. queueDeclare (queue: "task_queue", durable: true, exclusive: false, autoDelete: false, arguments: null); var message = G EtMessage (args); var body = Encoding. UTF8.GetBytes (message); var properties = channel. createBasicProperties (); properties. setPersistent (true); channel. basicPublish (exchange: "", routingKey: "task_queue", basicProperties: properties, body: body); Console. writeLine ("[x] Sent {0}", message);} Console. writeLine ("Press [enter] to exit. "); Console. readLine ();} private static string GetMessage (string [] Args) {return (args. Length> 0 )? String. Join ("", args): "Hello World! ");}}View Code

Worker. cs:

Using System; using RabbitMQ. client; using RabbitMQ. client. events; using System. text; using System. threading; class Worker {public static void Main () {var factory = new ConnectionFactory () {HostName = "localhost"}; using (var connection = factory. createConnection () using (var channel = connection. createModel () {channel. queueDeclare (queue: "task_queue", durable: true, exclusive: false, autoDelete: false, arguments: null); channel. basicQos (prefetchSize: 0, prefetchCount: 1, global: false); Console. writeLine ("[*] Waiting for messages. "); var consumer = new EventingBasicConsumer (channel); consumer. stored ed + = (model, ea) =>{ var body = ea. body; var message = Encoding. UTF8.GetString (body); Console. writeLine ("[x] Received {0}", message); int dots = message. split ('. '). length-1; Thread. sleep (dots * 1000); Console. writeLine ("[x] Done"); channel. basicAck (deliveryTag: ea. deliveryTag, multiple: false) ;}; channel. basicConsume (queue: "task_queue", noAck: false, consumer: consumer); Console. writeLine ("Press [enter] to exit. "); Console. readLine ();}}}View Code

RabbitMQ customer API online reference: RabbitMQ. NET client API reference online.