Solve RABBITMQ message loss and recurring consumption problem

Original address: http://blog.csdn.net/yeweiouyang/article/details/74943278

This article only records troubleshooting and problem locating, resolving the process. 1. Background

Recent user feedback submitted SQL query has been in a long waiting state, after the investigation, found that some of the query requests are missing, causing the user to submit queries are not received normally, and then long time no response.

Symptom: Market SQL Console submit 10 simple SQL queries, message sender: Send 10 messages to Message Queuing--message consumers: consume only 7 messages 2. Status Quo 2.1. The overall flow of the current SQL query

Producer: PHP:
Records the user's SQL query in the DB table, identifies the query task status (F_status) as running, sends information such as the task ID, submitter, and so on in the DB table to RabbitMQ; Message Queuing: RabbitMQ:
The PHP message is submitted to the switch; the switch then distributes the message to the specified message queue; Consumer: Python:
The main process listens to the message queue, and once a message is pulled, a message is pulled, a idle process is picked up from the process pool to process the message, and then the feedback is ACK to the message queue, and the message is removed from the message queue; 2.2. Message sender: Web side

Conclusion: The message is sent normally
Troubleshooting steps: View Log 2.3. Message Queuing

Conclusion: The number of messages is normal
Diagnostic steps:
Execution machine Installation Rabbitmq-dump-queue plug-in, used for dump queue messages;
1. Execution machine: stop service;
2. User: Submit 10 SQL queries:
3. Sender: Check the output log of the Web service side to determine that 10 messages have been written to the message queue;
4. Execution machine: Check the queue message through Rabbitmq-dump-queue, confirm the normal 10 messages are written;

Watch-n 1 ' $GOPATH/src/rabbitmq-dump-queue/rabbitmq-dump-queue-uri= "amqp://guest:guest@xxxxx:5672"-queue Ph_open_ Task

5. Execution machine: Start the service, messages in the message queue are all received; 2.4. Message Receivers

Code logic:

Try:pool = Pool (processes=40) def callback (ch, method, properties, body): try:dosomething.
            . Pool.apply_async (process) except Exception as E:print traceback.format_exc () logger_msg.i NFO (Traceback.format_exc ()) Finally://There will be a problem, even if the message is not processed will be feedback ack to RABBITMQ ch.basic_ack (deliv Ery_tag=method.delivery_tag) While True:try:connection = Pika. Blockingconnection (Pika. Connectionparameters (host= ' xxxxxxxx ')) Channel = Connection.channel () channel.queue_declare (queue= Queue_name, Durable=true) Channel.basic_qos (prefetch_count=1) Channel.basic_consume (callback, Queu
            E=queue_name, No_ack=false) channel.start_consuming () except pika.exceptions.ConnectionClosed as E: Continue except Exception as E:logger_msg.info (Traceback.format_exc ()) Finally:channel.basic_ack (Deliv ErY_tag=method.delivery_tag) Pool.close () Pool.join () 

Conclusion: In this example, the consumer master process will continue to listen to MQ, once MQ has a message will be pulled, and then start a child process from the process pool to process the message, but the process from the process pool to start the child process is not necessarily successful (if the current process pool does not have idle child processes), and the main process in any case, the Thus MQ removes the unhandled message, causing the message to be lost 3. Scenario

The problem arises in the consumer segment, so changes to consumers need to adjust the structure of the consumer:
* Original logic: Using Process Pool technology, the main process is responsible for listening, receiving MQ messages, sub-process is responsible for the implementation of MQ messages, the disadvantage is that a single main process can not easily handle the ACK status code, difficult to maintain;
* Existing logic: Using RABBITMQ's own characteristics (work_queue), consumers no longer maintain process pool, is a single process, responsible for listening, receiving, processing MQ messages, after processing the feedback ACK status code, process and process between non-interference, easy to maintain, can increase the consumer process at any time when the concurrency is large;

Current solution issues and solutions:

Issue 1: Message repeat consumption
Description: When a user stops querying on a page, it causes the consumer process to be killed, so the ACK status code is not fed back to MQ, so the message persists in MQ and is re-consumed when the new consumer starts;
Solution: Each time the consumer executes the query, the first query the execution state of the task on the DB, if the "Cancel/Fail/success" indicates that it has been consumed by other consumers, then directly return the ACK status code to MQ and remove the message from MQ;

Issue 2: How the process pool is maintained.
Description: When the user stops the query, it causes the consumer process to be killed, resulting in a decrease in the number of consumers;
Solution: Maintain a monitoring script, polling the number of consumer processes per minute, if less than 40 processes, the new start a consumer, until the quantity is sufficient;