Replace RABBITMQ with MongoDB

Original: http://blog.nosqlfan.com/html/3223.html

RABBITMQ is widely used as a queuing service system, and its supporting clients and monitoring operation plan are also more mature. The Boxedice queue service has switched from RABBITMQ to MongoDB since April this year, and has been running steadily so far, the following is a discussion ppt and related narration of Boxedice queue system. Share to everyone.

Why use a queuing system?

• Because some tasks need to be performed in the background, the caller does not have to wait for it to complete to return. For example, to send users mail, SMS and other operations.
• There is also a large system internal communication, may be queued to pass the message.

What are the requirements for the queuing system?

• Dealing with task handlers: usually some processes get messages from the queue for processing, and usually these processes start a lot. So the queue needs to be able to handle concurrent data request operations.
• Atomicity: Elements in a queue can only be fetched once, and it is necessary to ensure that the elements in each read queue are manipulated and deleted in an atomic nature.
• Fast: The queuing system is able to quickly process the write and read operations of elements.
• Garbage collection capability: If a task is half dead, you must have a way to monitor it and put the task back in the queue.

About Task handling

The task processing process works by reading the message from the queue and processing the message. So it needs an ability to take elements out of the queue and handle them. RABBITMQ provides the AMQP protocol, there are already many clients based on this protocol, and, in contrast, MongoDB has rich client support based on its MONGO wire protocol protocol.

Boxedice is used when using RABBITMQ Pika client, the conversion to MongoDB is used Pymongo. The two protocols are relatively much less expensive than the Pymongo.

For Atomicity, RABBITMQ is implemented through support for the Consume/ack protocol. And since MongoDB supports only the atomic personality of a single document, you can use its findandmodify command with the following simple syntax:

Db.runcommand ({findandmodify:collection, {options}})

The options here are an array that contains some of the following elements:

• where: is a query condition, for example, in our case, the query condition is {' where ' {' InProg ': false, ' done ': false}} The operation will be for the first element of the query.
• Sort: Is the ordering of the query structure, which sets the criteria by which the query returns results. For example, you can set a message priority, and then sort by the descending order of priority, so that messages with high priority are processed first.
• Update: This is the element that indicates how you need to personalize this, in our example, we set its InProg ID to true, indicate that it is being processed, set the processing time t to the current time, and this time will be used in garbage collection. as follows: {' update ': {' $set ': {' InProg ': true, ' t ': New Date ()}}}

Garbage collection

When we process the message above, we set its InProg to true to indicate that it is being processed, and then set done to True when the successful processing is complete, but if there is a problem during processing, a task with a inprog of true but done will always be false. The garbage collection strategy needs to be processed by checking the processing time t for garbage collection.

now = Datetime.datetime.now () difference = Datetime.timedelta (seconds=10) timeout = now-differencequeue.find ({' InProg ' : True, ' start ': {' $lte ': Timeout}})

For example, the above code, we determine whether T is the current time 10 seconds ago to determine if the expiration (10 seconds are not processed, we think the task processing failed), these failed messages we can do related processing, let it rejoin the message queue.

Some of the other considerations

In addition to the above mentioned speed, atomicity and other characteristics, for a queue system, there are some other aspects to consider.

• Fault tolerance: MongoDB's replica sets architecture provides overall high availability. When it is used as a queue, it also inherits this me. And RABBITMQ does not have built-in support. There is currently support for
• consistency in RABBITMQ 2.6.0: MongoDB defaults to flush data to disk in one minute, but it also provides a default 100ms operational log to enhance the reliability of its stand-alone. Can alleviate data loss caused by downtime. If you have a very high consistency requirement, you can also use MongoDB's GetLastError command to ensure that each operation is written to the operation log or on disk to return success.
• Extensibility: We use capped collection to do Message Queuing, so the purge of old data is automatic. In MongoDB, the data can be scaled out in a sharding manner, but sharding is not supported for capped collection. You can choose the way you want your application.