Using the RABBITMQ series two-task queue in node. js

In the previous article using the RABBITMQ Series in node. js a Hello world I used a task queue, but the scene at the time was to send a message to a consumer, and in this article I'll discuss scenarios with multiple consumers.

In fact, the problem with the core of the task queue is to avoid dealing with time-consuming tasks immediately, that is, the synchronous pattern of avoiding request-response. Instead, we use a scheduling algorithm that allows these time-consuming tasks to be executed later, in the form of asynchronous patterns. We need to encapsulate a message into a task and add it to the task queue. The background will run multiple worker processes (worker process), through the scheduling algorithm, the queue of tasks in sequence, and to one of the work process to perform processing. This concept is especially useful for Web applications where HTTP is short-connected, and they are not able to handle this complex task in a short period of time.

Preparatory work

We treat string-type messages as time-consuming tasks, and each string message ends with some points. Each point represents the number of seconds that the task needs to be consumed. The settimeout function can be used to simulate when the worker process is processed. For example: a fake time-consuming task is hello., then this task consumes 1 seconds, a bogus time-consuming task is hello:, then this task consumes 2 seconds, a bogus time-consuming task is hello ..., and the task takes 3 seconds to process.

Here slightly to the previous article of the Send.js file modification, so that it can send a user-defined arbitrary message, the program will hand over the task queue, we use New_task.js to name:

var q = ‘task_queue‘;var msg = process.argv.slice(2).join(‘ ‘) || "Hello World!";ch.assertQueue(q, {durable: true});ch.sendToQueue(q, new Buffer(msg), {persistent: true});console.log(" [x] Sent ‘%s‘", msg);

Our Receive.js file also needs to be modified, which requires the processing time of the forgery task, which makes the string type of task appear to take a few seconds (depending on. Number of items).

ch.consume(q, function(msg) {  var secs = msg.content.toString().split(‘.‘).length - 1;  console.log(" [x] Received %s", msg.content.toString());  setTimeout(function() {    console.log(" [x] Done");  }, secs * 1000);}, {noAck: true});

As an example of a worker worker process, let's start with a worker process on the left Shell window, then send a custom message to the right shell window and perform a look at the effect:

Polling scheduling algorithm (Round-robin)

One of the advantages of using task queues is the ability to execute tasks in parallel, and if you need to handle a lot of backlogs, we just need to add more workers as we run the worker process above, which makes scalability much easier.
First, we use ITEM2 to open 2 shell windows and run two worker processes inside, but which worker will handle the message? Here we can do a simple experiment to see, as shown, the left side is a two worker process, the right side is the message sender:

By default, RABBITMQ uses the Round-robin algorithm to distribute tasks in the task queue, sending the task to the next consumer each time it is distributed, so that the number of tasks that each consumer (worker process) handles is actually as large.

Message Confirmation

It takes a long time to handle a complex task, and in this time period, it's possible that our worker process has been hung up for some reason, and this anomaly needs to be considered. But we do not have this exception in our existing code, and when RABBITMQ sends the task to the worker process, we immediately remove the task from memory, assuming that the worker receives the message, we immediately kill the process, The task was not successfully executed at this time. At the same time, we also lose all of the task information distributed to this worker process but not yet processed.
However, we do not want to lose any of the tasks, if a worker process hangs, we would prefer to be able to send this task to other workers to deal with.
To avoid the loss of task information, RABBITMQ supports message acknowledgement. After a task is sent to the worker process and successfully processed, an ACK (message acknowledgement) identity is sent back from the consumer to tell RABBITMQ that the task has been processed and can be deleted.
If a consumer hangs up (a common reason such as the message channel is closed, the connection is lost, the TCP connection is lost), no message is sent to RABBITMQ to confirm the ACK's identity, and this time RABBITMQ will add it to the queue, and if there are other consumers, Then RABBITMQ will re-distribute the task immediately. In the previous example we did not open the message to confirm this option, and now we can open it with {Noack:false}:

ch.consume(q, function(msg) {  var secs = msg.content.toString().split(‘.‘).length - 1;  console.log(" [x] Received %s", msg.content.toString());  setTimeout(function() {    console.log(" [x] Done");    ch.ack(msg);  }, secs * 1000);}, {noAck: false});     // 开启消息确认标识

You can use CTRL + C to do an experiment to see the effect.

Message persistence

Just talked about if a worker process hangs up and does not let the message get lost in practice. But what if the entire RABBITMQ server hangs out? When a RABBITMQ service exits or is interrupted, it forgets the message in the task queue unless you tell it not to throw it away, that is, we notify the RABBITMQ task queue and these tasks need to be persisted.

First, we need to make sure that RABBITMQ never loses our task queue.

ch.assertQueue(‘hello‘, {durable: true});

However, you will find that this does not work because the Hello queue we have defined and specifies that it does not need to be persisted. RABBITMQ does not allow us to redefine the existing task queue by changing the configuration of the parameters, so we need to define a new task queue.

ch.assertQueue(‘task_queue‘, {durable: true});

This line of code needs to be modified at the same time as the relevant code within the producer and consumer.

Next, we need to configure the persistent option so that the messages we send are also persisted.

ch.sendToQueue(q, new Buffer(msg), {persistent: true});

Fair Dispatch

In the previous example, we discussed the scheduling of RABBITMQ, which uses the Round-robin polling scheduling algorithm, so it distributes the messages evenly to each worker process. RABBITMQ does not care how many messages per worker process are not confirmed, it will only keep you dispatched, whether you can handle it or not. This time, the problem arises, assuming that there are 2 workers, of which 1 workers are unfortunately assigned a very complex task, it may take several hours, the other worker is assigned to the task is relatively simple, only a few minutes to complete the process, Due to the RABBITMQ assignment problem, many new tasks will still be assigned to the worker who is working on the time-consuming task, and the task behind the worker will be waiting. Fortunately, RABBITMQ can designate a worker with prefetch (1) to distribute up to 1 tasks at the same time, and a new task will be dispatched once the task processing has been sent with a confirmation notification.

ch.prefetch(1);

The final code

New_task.js Code:

var amqp = require(‘amqplib/callback_api‘);amqp.connect(‘amqp://localhost‘, function(err, conn) {  conn.createChannel(function(err, ch) {    var q = ‘task_queue‘;    var msg = process.argv.slice(2).join(‘ ‘) || "Hello World!";    ch.assertQueue(q, {durable: true});    ch.sendToQueue(q, new Buffer(msg), {persistent: true});    console.log(" [x] Sent ‘%s‘", msg);  });  setTimeout(function() { conn.close(); process.exit(0) }, 500);});

Worker.js:

var amqp = require(‘amqplib/callback_api‘);amqp.connect(‘amqp://localhost‘, function(err, conn) {  conn.createChannel(function(err, ch) {    var q = ‘task_queue‘;    ch.assertQueue(q, {durable: true});    ch.prefetch(1);    console.log(" [*] Waiting for messages in %s. To exit press CTRL+C", q);    ch.consume(q, function(msg) {      var secs = msg.content.toString().split(‘.‘).length - 1;      console.log(" [x] Received %s", msg.content.toString());      setTimeout(function() {        console.log(" [x] Done");        ch.ack(msg);      }, secs * 1000);    }, {noAck: false});  });});

Using the RABBITMQ series two-task queue in node. js