[RabbitMQ] 5. RabbitMQ task distribution mechanism and rabbitmq

[RabbitMQ] 5. RabbitMQ task distribution mechanism and rabbitmq

When a Consumer requires a large number of operations, the RabbitMQ Server needs a certain distribution mechanism to balance the load of each Consumer. Next we will explain the distribution.

The application scenario is that the RabbitMQ Server will distribute queue messages to different consumers to process computing-intensive tasks.

1. Message acknowledgment Message confirmation

Each Consumer may take some time to process the received data. If a Consumer error occurs during this process, the exception exits, and the data has not been processed, it is unfortunate that the data is lost. Because we use the no-ack Method for confirmation, that is to say, every time the Consumer receives the data, regardless of whether the data is processed, the RabbitMQ Server immediately marks the Message as complete, then deleted from the queue.

If a Consumer exits unexpectedly, the data it processes can be processed by another Consumer, so that the data will not be lost in this case (note that in this case ).

To ensure that data is not lost, RabbitMQ supports message confirmation, that isAcknowledgments. To ensure that the data can be correctly processed and not only received by Consumer, we cannot use no-ack. Instead, an ack should be sent after the data is processed.

The ack sent after processing the data tells RabbitMQ that the data has been received and processed completely. RabbitMQ can safely delete the data.

If the Consumer exits but does not send ack, RabbitMQ will send the Message to the next Consumer. This ensures that data will not be lost when the Consumer exits abnormally.

The timeout mechanism is not used here. RabbitMQ only confirms that the Message is not correctly handled through connection interruption of Consumer. That is to say, RabbitMQ gives Consumer enough time to process data.

In this way, even if you interrupt Recieve. cs through Ctr-C, the Message will not be lost and it will be distributed to the next Consumer.

If you forget ack, the consequences are very serious. When the Consumer exits, the Message is reselected. Then RabbitMQ will occupy more and more memory. Because RabbitMQ will run for a long time, this "Memory leakage" is fatal. To debug this error, run the following command to print un-acked Messages.

2. Round-robin dispatching

The distribution mechanism of RabbitMQ is very suitable for expansion, and it is specially designed for concurrent programs. If load increases, you only need to create more Consumer for task processing. Of course, what should I do if I need to increase the load? I have never encountered this situation, so I can create multiple virtual hosts to refine the communication categories.

1. enable two consumers, that is, run two Recieve. cs.

2. When two Producer instances are enabled, two Producer. cs instances are run.

By default, RabbitMQ distributes each Message sequentially. After each ack is received, the Message is deleted and the next Message is distributed to the next Consumer. This distribution method is called round-robin (elegant distribution ).

Producer. cs

class Program    {        static void Main(string[] args)        {            ConnectionFactory factory = new ConnectionFactory() { HostName = "localhost" };            using (IConnection connection = factory.CreateConnection())            {                using (IModel channel = connection.CreateModel())                {                    channel.QueueDeclare("hello", false, false, false, null);                    var message = GetMessage(args);                    var body = Encoding.UTF8.GetBytes(message);                                        var properties = channel.CreateBasicProperties();                    properties.DeliveryMode = 2;//non-persistent (1) or persistent (2)                     //channel.TxSelect();                    channel.BasicPublish("", "hello", properties, body);                    //channel.TxCommit();                }            }        }        private static string GetMessage(string[] args)        {            return ((args.Length > 0) ? string.Join(" ", args) : "Hello World!");        }    }

Consumer. cs

// # Define demo1 # define demo2using RabbitMQ. client; using RabbitMQ. client. events; using System. collections. generic; using System. linq; using System. text; using System. threading; using System. threading. tasks; namespace ReceiveDemo2 {// <summary> /// one Send and multiple Receive examples, /// an ack example is added. /// elegant distribution // </summary> class Program {static void Main (string [] args) {var factory = new ConnectionFactory () {HostName = "localhost "}; using (var connection = factory. createConnection () {using (var channel = connection. createModel () {channel. queueDeclare ("hello", false, null); var consumer = new QueueingBasicConsumer (channel); # if demo1 channel. basicConsume ("hello", true, consumer); // automatically deletes the message # else channel. basicConsume ("hello", false, consumer); // The receiver needs to send an ack receipt and delete the message # endif Console. writeLine ("[*] Waiting for messages. "+" To exit press CTRL + C "); while (true) {var ea = (BasicDeliverEventArgs) consumer. queue. dequeue (); // pending operation # if demo2 channel. basicAck (ea. deliveryTag, false); // with channel. basicConsume ("hello", false, null, consumer); # endif var body = ea. body; var message = Encoding. UTF8.GetString (body); Console. writeLine ("[x] Received {0}", message); int dots = message. split ('. '). length-1; Thread. sleep (dots * 1000); Console. writeLine ("[x] Done"); # if demo2 // channel. basicAck (ea. deliveryTag, false); // with channel. basicConsume ("hello", false, null, consumer); accordingly, the running results of lines 40 and 49 are different. if [x] terminated ed {0} is output and [x] Done is not output, CTRL + C ends the program, the message is automatically sent to another client, after the other client receives the message and runs the 49-line command, the server deletes the message # endif }}}}}}

3. Message durability Message persistence

In the previous section, we learned that even if the Consumer exits unexpectedly, the Message will not be lost. But what if the RabbitMQ Server exits? Software has bugs, even if the RabbitMQ Server is perfect and has no bugs (of course, this is impossible. It is software with bugs. If there is no bug, it is not software ), it is still possible to exit: it is affected by other software, or the system is restarted, and the system panic is gone...

To ensure that data is not lost after RabbitMQ exits or crash ends, both queue and Message must be persistent. For the persistence of queue, you must specify durable = True at the time of declaration to modify the channel of Producer and Consumer. the QueueDeclare code re-emphasizes that both Producer and Consumer should create this queue, even though creation in only one place actually works:

bool durable = true;channel.QueueDeclare("hello", durable, false, false, null);

There will be no errors in the preceding statement execution, but we cannot get the expected results because the RabbitMQ Server has maintained a queue named hello, so the above execution will not be of any effect, that is, no attributes of hello will be affected. This is also discussed in the previous article.

Workaround is also very simple. Declare another name named queue, for example, positioning task_hello by name, or through monitoringHttp: // localhost: 15672/, Delete the Queue named "hello.

Next, you also need to specify a properties when Publish is in Producer. cs by using the following method:

Static void Main (string [] args) {var factory = new ConnectionFactory () {HostName = "localhost"}; using (var connection = factory. createConnection () {using (var channel = connection. createModel () {bool durable = true; channel. queueDeclare ("task_queue", durable, false, false, null); // you must specify durable = True var message = GetMessage (args) when declaring the persistence of queue ); var body = Encoding. UTF8.GetBytes (message); var properties = channel. createBasicProperties (); properties. setPersistent (true); // persistent Message is required, that is, a properties, channel, is specified in Publish. basicPublish ("", "task_hello", properties, body );}}}

Further discussion on Persistence:

To avoid data loss, we adopt:

But can this ensure that 100% of the data is not lost?

The answer is no. The problem is that it takes time for RabbitMQ to store the information on the disk. Although this time window is short, it does exist. If the data is not saved in this time window, the data will be lost. Another reason is that RabbitMQ does not perform fsync for every Message: it may just save it to the Cache and it has not been saved to the physical disk.

Therefore, this persistence still has a problem. But for most applications, this is enough. To maintain consistency, you can put the publishTransaction. The implementation of this transaction requires user defined codes.

So what will commercial systems do? One possible solution is to set aside time for each application to flash cache when the system panic is running or when the system is restarted abnormally or when the power is down, so that each application can exit gracefully.

4. Fair dispatch Fair distribution

If a Consumer has a heavy workload, some Consumer will be able to do nothing, and some Consumer will have no chance to rest. So how does RabbitMQ solve this problem?

Set prefetchCount = 1 using the BasicQos method. In this way, RabbitMQ enables each Consumer to process at most one Message at the same time point. In other words, it will not distribute the new Message to the Consumer before receiving the ack of the Consumer. The setting method is as follows:

channel.BasicQos(0, 1, false);

Note that this method may cause the queue to be full. Of course, in this case, you may need to add more Consumer or create more virtualHost to refine your design.

Consumer. cs

Static void Main (string [] args) {var factory = new ConnectionFactory () {HostName = "localhost"}; using (var connection = factory. createConnection () {using (var channel = connection. createModel () {bool durable = true; channel. queueDeclare ("task_queue", durable, false, false, null); channel. basicQos (0, 1, false); // in this way, RabbitMQ enables each Consumer to process at most one Message at the same time point. In other words, it will not distribute the new Message to the Consumer before receiving the ack of the Consumer. Var consumer = new QueueingBasicConsumer (channel); channel. basicConsume ("task_hello", false, null, consumer); // The receiver needs to send an ack receipt to delete the message Console. writeLine ("[*] Waiting for messages. "+" To exit press CTRL + C "); while (true) {var ea = (BasicDeliverEventArgs) consumer. queue. dequeue (); // pending Operation channel. basicAck (ea. deliveryTag, false); // with channel. basicConsume ("task_queue", false, null, consumer); var body = ea. body; var message = Encoding. UTF8.GetString (body); Console. writeLine ("[x] Received {0}", message); int dots = message. split ('. '). length-1; Thread. sleep (dots * 1000); Console. writeLine ("[x] Done ");}}}}

Source: https://www.cnblogs.com/qiyebao/p/4205626.html