Getting started with RabbitMQ-working queue and rabbitmq

Getting started with RabbitMQ-working queue and rabbitmq

What is a work queue?

A work queue is a processing method to avoid waiting for a number of resources or time-consuming operations. We encapsulate the task as a message and send it to the queue. The consumer keeps pulling the task from the backend and executing it. When multiple consumer worker processes are run, tasks in the queue are shared among each consumer.

The advantage of working queue is that it can process tasks in parallel. If many tasks are accumulated in the queue, you only need to add more consumption, which is very convenient to expand.

Preparations

1. Create producer and consumer clients

2. Use Thread. Sleep () in the consumer to simulate time-consuming operations

 

ProducerTaskQueuesProducer. cs

 

Using RabbitMQ. Client;

Using System;

Using System. Collections. Generic;

Using System. Diagnostics;

Using System. Linq;

Using System. Text;

Using System. Threading. Tasks;

 

Namespace RabbitMQProducer

{

Public class TaskQueuesProducer

{

Static int processId = Process. GetCurrentProcess (). Id;

Public static void Send ()

{

Console. WriteLine ($ "I Am a producer {processId }");

Var factory = new ConnectionFactory ()

{

HostName = "127.0.0.1"

};

Using (var connection = factory. CreateConnection ())

{

Using (var channel = connection. CreateModel ())

{

Channel. QueueDeclare (queue: "taskqueue", durable: false, exclusive: false, autoDelete: false, arguments: null );

For (int item = 0; item <20; item ++)

{

String message = $ "message sent by the producer {processId}: {item }";

Channel. BasicPublish (exchange: "", routingKey: "taskqueue", basicProperties: null, body: Encoding. UTF8.GetBytes (message ));

Console. WriteLine (message );

}

 

Console. WriteLine ("Press [enter] to exit .");

Console. ReadLine ();

}

}

}

}

}

 

ConsumerTaskQueuesConsumer. cs

 

Using System;

Using System. Collections. Generic;

Using System. Linq;

Using System. Text;

Using System. Threading. Tasks;

Using RabbitMQ. Client;

Using RabbitMQ. Client. Events;

Using System. Diagnostics;

Using System. Threading;

 

Namespace RabbitMQConsumer

{

Public class TaskQueuesConsumer

{

Static int processId = 0;

Static TaskQueuesConsumer ()

{

ProcessId = Process. GetCurrentProcess (). Id;

}

Public static void Receive ()

{

Console. WriteLine ($ "I Am a consumer {processId }");

Var factory = new ConnectionFactory ()

{

HostName = "127.0.0.1"

};

Using (var connection = factory. CreateConnection ())

{

Using (var channel = connection. CreateModel ())

{

Channel. QueueDeclare (queue: "taskqueue", durable: false, exclusive: false, autoDelete: false, arguments: null );

EventingBasicConsumer consumer = new EventingBasicConsumer (channel );

Consumer. Received + = Consumer_Received;

// Noack = false if the message is not automatically confirmed, you must manually call channel. BasicAck () to confirm the message.

// Noack = true automatic message confirmation. When the message is sent to the consumer (consumers) by RabbitMQ, it will be immediately removed from the memory.

Channel. BasicConsume (queue: "taskqueue", noAck: false, consumer: consumer );

 

Console. WriteLine ("Press [enter] to exit .");

Console. ReadLine ();

}

}

}

 

Private static void Consumer_Received (object sender, BasicDeliverEventArgs e)

{

String message = Encoding. UTF8.GetString (e. Body );

Console. WriteLine ($ "received message: {message }");

// Thread. Sleep (new Random (). Next (1000,100 0*5); // Simulate message processing time-consuming operations

Console. WriteLine ($ "message processed ");

 

// Send the message confirmation receipt corresponding to noack = false in BasicConsume.

// EventingBasicConsumer consumer = sender as EventingBasicConsumer;

// Consumer. Model. BasicAck (e. DeliveryTag, false );

}

}

}

 

When we run three consumer clients and one producer client, we find that the 20 messages sent by the producer are evenly received and processed by the three clients.

This is the default message distribution mechanism of RabbitMQ-round robin. By default, RabbitMQ sends messages to each consumer in order, and each consumer receives the same number of messages on average.

 

Message confirmation

In the current Code, when a message is sent to consumer by RabbitMQ, it will be deleted immediately. In this case, if one of the consumer clients is stopped, the messages being processed will be lost, at the same time, all messages sent to this worker and not processed will be lost. This is not what we want to see. We hope that if a consumer client fails, we want to resend the task to another consumer client.

To prevent message loss, RabbbitMQ provides a message confirmation mechanism. The consumer uses an ack to notify RabbitMQ that a message has been received and processed, and then RabbitMQ releases and deletes the message.

If the consumer fails and no response is sent, RabbitMQ considers that the message is not processed and resends the message to other consumers, so that even if a consumer fails, the message will not be lost.

There is no supermarket concept for a message. When a worker disconnects from it, RabbitMQ resends the message so that it will not cause problems when processing long tasks.

In the previous code, we enabled automatic message confirmation. Once the consumer fails, the message will be lost. Now let's modify the two codes to enable the message confirmation mechanism.

 

Set noack to false to disable automatic message confirmation.

 

Channel. BasicConsume (queue: "taskqueue", noAck: false, consumer: consumer );

 

Uncomment the following code to confirm receipt

 

EventingBasicConsumer consumer = sender as EventingBasicConsumer;

Consumer. Model. BasicAck (e. DeliveryTag, false );

 

Note: Once you forget the message confirmation, the message will be re-sent after your program is launched. If you cannot release the unresponded message, RabbitMQ will occupy more and more memory.

You can run the following command to check the message information that you forgot to confirm or view it on the RabbitMQWebweb Management page.

 

Rabbitmqctl list_queues name messages_ready messages_unacknowledged

 

After the modification is completed, run it again, so you don't have to worry about message loss.

 

Message persistence

Without special settings, all queues and messages will be lost when the RabbitMQ service is closed or crashed. To ensure that messages are not lost, we need to do two things: Set the queue and message to persistent.

Set the queue to persistent. both producer and consumer must be modified.

 

Channel. QueueDeclare (queue: "taskqueue", durable: true, exclusive: false, autoDelete: false, arguments: null );

Set message persistence

 

Var properties = channel. CreateBasicProperties ();

Properties. DeliveryMode = 2; // delivery mode of the DeliveryMode message. The default value is 1, which is non-persistent. DeliveryMode = 2 stores the message content persistently.

Channel. BasicPublish (exchange: "", routingKey: "taskqueue", basicProperties: properties, body: Encoding. UTF8.GetBytes (message ));

 

Note:

Fair Scheduling

RabbitMQ only distributes messages that enter the queue, but does not care about the busy or idle consumer. This can easily lead to busy consumer, idle, and cannot maximize the use of resources.

To solve this problem, RabbitMQ provides the basicQos method, and the transfer parameter is prefetchCount = 1. This tells RabbitMQ not to send more than one message to a consumer at the same time. That is, the next message is sent only when the consumer is idle.

 

Channel. BasicQos (prefetchSize: 0, prefetchCount: 1, global: false );

 

After the preceding settings are executed, you will find that the message is not forwarded according to the previous Round robin (Round-robin), but is forwarded only when the consumer is not busy.

Because the message is not sent out, the consumer can be immediately put into work after the consumer is added dynamically. The default round robin forwarding mechanism does not support dynamic addition of consumers because the message has been allocated, cannot join the job immediately, even if there are many unfinished tasks.

Note:

This method may cause the queue to be full. Of course, in this case, you may need to add more Consumer or create more virtualHost to refine your design.

Complete code

 

ProducerTaskQueuesProducer. cs

 

Using RabbitMQ. Client;

Using System;

Using System. Collections. Generic;

Using System. Diagnostics;

Using System. Linq;

Using System. Text;

Using System. Threading. Tasks;

 

Namespace RabbitMQProducer

{

Public class TaskQueuesProducer

{

Static int processId = Process. GetCurrentProcess (). Id;

Public static void Send ()

{

Console. WriteLine ($ "I Am a producer {processId }");

Var factory = new ConnectionFactory ()

{

HostName = "127.0.0.1"

};

Using (var connection = factory. CreateConnection ())

{

Using (var channel = connection. CreateModel ())

{

// Durable: Persistent storage queue

// AutoDelete: automatic deletion. If the queue does not have any subscribed consumers, the queue will be automatically deleted. This type of queue applies to temporary queues.

// Exclusive: exclusive queue. If a queue is declared as an exclusive queue, the queue is only visible to the connection that is declared for the first time and automatically deleted when the connection is disconnected. Note: 1. The exclusive queue is visible Based on connections. Different channels of the same connection can access the exclusive queue created by the same connection at the same time. 2. For the first time, if an exclusive queue has been declared for a connection, other connections are not allowed to establish an exclusive queue with the same name. This is different from a common queue. 3. Even if the queue is persistent, the exclusive queue is automatically deleted once the connection is closed or the client exits. This queue is applicable to scenarios where only one client sends and reads messages.

Channel. QueueDeclare (queue: "taskqueue", durable: true, exclusive: false, autoDelete: false, arguments: null );

For (int item = 0; item <200000; item ++)

{

String message = $ "message sent by the producer {processId}: {item }";

Var properties = channel. CreateBasicProperties ();

Properties. DeliveryMode = 2; // delivery mode of the DeliveryMode message. The default value is 1, which is non-persistent. DeliveryMode = 2 stores the message content persistently.

Channel. BasicPublish (exchange: "", routingKey: "taskqueue", basicProperties: properties, body: Encoding. UTF8.GetBytes (message ));

Console. WriteLine (message );

}

 

Console. WriteLine ("Press [enter] to exit .");

Console. ReadLine ();

}

}

}

}

}

 

ConsumerTaskQueuesConsumer. cs

 

Using System;

Using System. Collections. Generic;

Using System. Linq;

Using System. Text;

Using System. Threading. Tasks;

Using RabbitMQ. Client;

Using RabbitMQ. Client. Events;

Using System. Diagnostics;

Using System. Threading;

 

Namespace RabbitMQConsumer

{

Public class TaskQueuesConsumer

{

Static int processId = 0;

Static TaskQueuesConsumer ()

{

ProcessId = Process. GetCurrentProcess (). Id;

}

Public static void Receive ()

{

Console. WriteLine ($ "I Am a consumer {processId }");

Var factory = new ConnectionFactory ()

{

HostName = "127.0.0.1"

};

Using (var connection = factory. CreateConnection ())

{

Using (var channel = connection. CreateModel ())

{

Channel. QueueDeclare (queue: "taskqueue", durable: true, exclusive: false, autoDelete: false, arguments: null );

// Set prefetchCount = 1 in the BasicQos method. In this way, RabbitMQ enables each Consumer to process at most one Message at the same time point. In other words, before receiving the ack of the Consumer, it will not distribute the new Message to it.

Channel. BasicQos (prefetchSize: 0, prefetchCount: 1, global: false );

EventingBasicConsumer consumer = new EventingBasicConsumer (channel );

Consumer. Received + = Consumer_Received;

// Noack = false if the message is not automatically confirmed, you must manually call channel. BasicAck () to confirm the message.

// Noack = true automatic message confirmation. When the message is sent to the consumer (consumers) by RabbitMQ, it will be immediately removed from the memory.

Channel. BasicConsume (queue: "taskqueue", noAck: false, consumer: consumer );

 

Console. WriteLine ("Press [enter] to exit .");

Console. ReadLine ();

}

}

}

 

Private static void Consumer_Received (object sender, BasicDeliverEventArgs e)

{

String message = Encoding. UTF8.GetString (e. Body );

Console. WriteLine ($ "received message: {message }");

Thread. Sleep (new Random (). Next (1000,100 0*5); // simulates the time-consuming operations of message processing.

Console. WriteLine ($ "message processed ");

 

// Send the message confirmation receipt corresponding to noack = false in BasicConsume.

EventingBasicConsumer consumer = sender as EventingBasicConsumer;

Consumer. Model. BasicAck (e. DeliveryTag, false );

}

}

}