MOONMQ: High-performance message queue with Go

This is a creation in Article, where the information may have evolved or changed.

Introduced

Moonmq is a high-performance Message Queuing system implemented with go, and is ready for use in our messaging push service and asynchronous tasks in various backgrounds.

In the design above, Moonmq mainly borrowed rabbitmq and rocketmq related ideas, but did a lot of subtraction, after all, I am not designed to be a very generic MQ.

noun explanation

• Publisher, message producer
• Consumer, message Consumer
• Broker, message Broker
• Queue, message store queues

Publisher sends a message to a named queue Msg,broker is responsible for storing the MSG inside the queue.

Consumer can focus on the queue you are interested in so that when there is a message in the queue, the broker will push the message to the consumer.

Push-Pull model

In Rocketmq, pull MSG is supported, while RABBITMQ supports push and pull Msg. MOONMQ only supports push MSG. The main considerations are as follows:

• When the consumer online, push is the most timely, because this time will definitely be able to put MSG push success.
• When consumer is offline, Broker saves offline messages, and when consumer is online, the broker still pushes offline messages in push mode.

In addition, because MOONMQ follow-up will support our message push system, if using pull model, hundreds of thousands of consumer regular pull, I am a little worried that Moonmq will be too much.

Message type

Moonmq the msg into direct and Fanout,fanout is a broadcast message, MOONMQ will send any consumer that subscribes to the queue for MSG push.

If the type of MSG is DIRECT,MOONMQ will be polled, select a consumer for MSG push.

Message priority

MOONMQ does not support message prioritization, it can be cumbersome to handle, and usually we do not need particularly granular priority control.

Coarse-grained priority control can be achieved in a simple way:

• Set Queue1,queue2,queue3 three queues, queue1 to handle the message with the highest priority, queue2 second, queue3 lowest
• When Publisher sends a message, it is sent to the specified queue on the priority level.
• We can have multiple consumer processing queue1 messages, such as 3, then with 2 processing queue2, 1 processing queue1, so that the priority control.

Message filtering

MOONMQ messages are filtered by routing key.

When Publisher sends msg to a particular queue, it can also specify the corresponding routing key, and only consumer receives the MSG if it is concerned that the queue also specifies the same routing key.

Ack

MOONMQ supports the ACK mechanism, and when the push of a msg to consumer, consumer must respond to a ACK,MOONMQ before the MSG push succeeds. If there is no ACK for a long time, MOONMQ will re-select a consumer to send again.

ACK can greatly guarantee the success rate of message push, but it will have an effect on the fast push of message, so MOONMQ also supports no ACK mode, in which the MOONMQ sends successful MSG, it is considered that push succeeds without waiting for ACK receipt.

Deferred message

This is not supported yet, and the follow-up is the case.

Timed messages

The difficulty is relatively large, will not realize

Agreement

The MOONMQ uses a protocol similar to ROCKETMQ, as follows:

|total length(4 bytes)|header length(4 bytes)|header json|body|total length = 4 + len(header json) + len(body)header length = len(header json)

In Moonmq, we use proto to define the protocol.

type Proto struct {    Method uint32 `json:"method"`    Fields map[string]string `json:"fields"`    Body []byte `json:"-"`}

Moonmq any protocol, we need to take the method, we use method to do the actual message processing.

Moonmq method Reference RABBITMQ, there are several types of methods:

• Synchronizing the request method, the client must wait for the corresponding response method after sending the request method, and in the process of waiting, it can also handle asynchronous method such as Push,error.
• Synchronize response method, corresponding to a specific request method.
• Async method, no wait after sending.

At this stage, MOONMQ supports the following synchronous method:

• Auth, Auth_ok
• Publish, Publish_o
• Bind, BIND_OK
• Unbind, UNBIND_OK

The following Async method is supported as well:

• Push
• Error
• Heartbeat
• Ack

Subsequent

This is just a brief introduction of MOONMQ, follow-up we will continue to improve MOONMQ, and strive to become a good MQ products.

MOONMQ code here HTTPS://GITHUB.COM/SIDDONTANG/MOONMQ, look forward to everyone's feedback.