Share an open-source distributed Message Queue (equeue) written in c #.

This topic describes a distributed Message Queue (equeue) implemented by the way when writing enode some time ago. The idea of this message queue is not what I came up with. Instead, I learned about Alibaba's rocketmq and used c # To implement a lightweight and simple version. On the one hand, you can write this queue to gain a deeper understanding of some common problems of message queues. On the other hand, you can also use it to integrate with enode to provide support for message transmission between commands and domain events in enode. Currently, on the. net platform, the most common message queue is Microsoft's MSMQ, and there are also. net clients like rabbitmq. These message queues are very strong and mature. However, after learning kafka and Alibaba's rocketmq (metaq in earlier versions and renamed rocketmq after metaq 3.0), I felt that the design philosophy of rocketmq attracted me deeply, because I can not only understand its ideas, but also learn its complete source code. However, rocketmq is written in java and does not exist yet.. so after learning the rocketmq design documentation and most of the code, I decided to write it out in c. Open-source project address: Workshop. You can also see how to use it in the enode project. In the EQUEUE message queue, a Topic is a topic. In a system, we can divide messages into topics so that we can send messages to different queue through topics. Under a topic of Queue, we can set multiple queue. Each queue is the message queue we usually call. Because Queue belongs to a specific topic completely, therefore, when we want to send a message, we always need to specify the topic to which the message belongs. Then, the equeue will be able to know the number of queue under the topic. But which queue does it send? For example, if a topic has four queue, Which queue should be sent when a message under this topic is sent? There must be a message routing process. Currently, when sending a message, you need to specify the topic corresponding to the message and an object-type parameter for routing. The equeue obtains all the queue based on the topic, and obtains the number of the queue to be sent based on the object parameter through the hash code and the number of the modulo queue. The process of routing a message is done by the sender of the message, that is, the producer. The reason for not doing so on the Message Server is that this allows users to decide how to route messages themselves, with greater flexibility. The Producer is the Producer of the message queue. We know that the essence of message queue is to implement the publish-subscribe mode, that is, the producer-consumer mode. The producer produces messages and the consumer consumes messages. Therefore, the Producer is used to produce and send messages. A Consumer is the Consumer of a message queue. A message can have multiple consumers. Consumer Group, which may be a new concept for everyone. The reason why a consumer group is created is to achieve the cluster consumption described below. A consumer group contains some consumers. If these consumers consume messages in clusters, these consumers consume messages in the group on average. The Broker in the broker equeue is responsible for transferring messages, that is, receiving messages sent by the producer, then persistently sending messages to the disk, and then receiving requests for pulling messages sent by the consumer, then pull the corresponding message to the consumer according to the request. Therefore, the broker can be understood as a Message Queue Server that provides message receiving, storage, and pulling services. It can be seen that the broker is the core of the equeue and cannot be mounted. Once the broker crashes, the producer and consumer cannot implement publish-subscribe. Cluster consumption refers to the consumption of a consumer in a consumer group to consume the queue in a topic on average. For more information about how to calculate the average value, see the architecture diagram below. Here we will briefly describe it in text. Assume that a topic has four queue and a consumer group. The group has four consumers, and each consumer is allocated to a queue under the topic, in this way, the queue under the average consumption topic is achieved. If a consumer group has only two consumers, each consumer consumes two queue. If there are three consumers, the first one consumes two queue, and the last two consume one queue, so as to achieve the average consumption as much as possible. Therefore, we can see that we should try to make the number of consumer under the consumer group consistent with the number of queue of the topic or a multiple relationship. In this way, the number of queue consumed by each consumer is always the same, so that the pressure on each consumer server is similar. The premise is that the number of messages in each queue under this topic is almost the same. This can be ensured by performing hash routing on messages based on a user-defined key. Broadcast consumption broadcast consumption refers to a consumer who subscribes to a topic and receives messages from all queue under the topic, regardless of the consumer group. Therefore, for broadcast consumption, consumer group has no practical significance. When consumer is instantiated, we can specify cluster consumption or broadcast consumption. Consumption progress (offset): when a consumer in a consumer group consumes a message in a queue, the equeue records the consumption position (offset) to know where the current consumption is. So that the consumer can continue consumption from this position after restart. For example, if a topic has four queue and a consumer group has four consumers, each consumer is allocated to one queue, and each consumer consumes messages in its own queue respectively. The equeue records the consumption progress of each consumer on Its queue separately, so that each consumer can know where to continue consumption next time after restart. In fact, maybe the consumer will not consume the queue after the next restart, but will consume it by other consumer in the group, because we have recorded the consumption location of this queue. Therefore, it can be seen that the consumption location is irrelevant to the consumer, and the consumption location is completely an attribute of the queue, which is used to record the current consumption location. In addition, a topic can be subscribed to by consumer in multiple consumer groups. Even if the consumer in different consumer groups consumes the same queue under the same topic, the consumption progress is stored separately. That is to say, the consumption of consumer in different consumer groups is completely isolated and independent from each other. Another point is that for cluster consumption and broadcast consumption, the consumption progress persistence is different. The consumption progress of cluster consumption is placed on the broker, that is, the Message Queue Server, the consumption progress of broadcast consumption is stored on the local consumer disk. The reason for this design is that for cluster consumption, a queue consumer may change because the number of consumers in a consumer group may increase or decrease, then, we will re-calculate the number of queue consumed by each consumer. What is this understandable? So when a change occurs to the queue's consumer, how does the new consumer know where to start consuming this queue? If the consumption progress of this queue is stored on the previous consumer server, it is very difficult to get the consumption progress because the server may have been suspended or dismounted, is possible. Because the broker always serves all the consumers, the queue of the subscribed topic is stored on the broker during cluster consumption, different consumer groups are isolated during storage to ensure that the consumption progress of different consumer groups is mutually affected. Then, for broadcast consumption, since a queue's consumer will not change, we do not need to let the broker store the consumption location, so it is saved on the consumer's own server. What is EQUEUE? Through this, We can intuitively understand the equeue. This figure is taken from the rocketmq design document. Because the design idea of equeue is completely the same as that of rocketmq, I used it. Each producer can send messages to a topic. When sending messages, the messages are sent to a specific queue according to a specific routing policy (which can be customized by the producer. Then, the consumer can consume messages in the queue under a specific topic. In, TOPIC_A has two consumers, both of which are in a group, so the queue under TOPIC_A should be consumed on average, but because there are three queue, therefore, the first consumer has two queue and the second consumer has one. For TOPIC_ B, because there is only one consumer, all queue under TOPIC_ B is consumed by it. All topic information, queue information, and messages are stored on the broker server. This is not reflected in. Focus on the relationships between producer, consumer, topic, and queue, rather than the deployment architecture of physical servers. Key questions 1. the communication between producer, broker, and consumer is implemented in c # and is generally deployed in the LAN. To achieve high-performance communication, we can use asynchronous socket for communication .. Net itself provides good support for asynchronous socket communication; we can also use zeromq to achieve high-performance socket communication. I wanted to directly use zeromq to implement the communication module, but later I learned about socket communication in. net and found that it was not difficult, so I realized it myself. The advantage of self-implementation is that I can define the Message Protocol by myself. Currently, this part of implementation code is in the ecommon basic class library and is an independent basic class library that can be used and is irrelevant to the business scenario. If you are interested, download the code. After some performance tests, we found that the performance of the communication module is still good. One broker and four producers send messages to the broker at the same time. The number of messages that can be sent per second is no problem, and more producers have not been tested yet. 2. Performance issues and fast reading of messages are the main considerations for message persistence. 1. First, messages on a broker do not need to be stored on the broker server until they are consumed. According to Alibaba rocketmq design, messages that have been consumed are deleted once a day by default. Therefore, we can understand that messages on the broker should not grow without limit, because messages will be deleted regularly. Therefore, you do not need to consider the issue that messages cannot be stored on a broker. 2. How to quickly persist messages? In general, I think there are two ways: 1) sequential Writing of disk files; 2) using nosql products with ready-made keys and values for storage; rocketmq currently uses its own file writing method, the difficulty of this method is that writing files is complicated because all messages are appended to the end of the file in sequence. Although the performance is very high, the complexity is also high. For example, all messages cannot be written in one file, after a file reaches a certain size, it needs to be split. Once split, there will be many problems. It is also complicated to read data after splitting. In addition, because the file is written in sequence, we also need to record the start position and length of each message in the file, so that when the consumer consumes the message, in order to get the message from the file according to the offset. In short, there are many issues to consider. If you use nosql to persist messages, you can save the trouble of writing files. You only need to care about how to match the Message key with the message offset in the queue. Another question is, should the information in the queue be persistent? First, you need to know what is put in the queue. After receiving a message, the broker must first persist the message and put the message in the queue. However, due to the limited memory, it is impossible for us to put this message directly into the queue. What we need to put is the key of the message in nosql, or if the object is used for persistence, the offset of the message in the file is placed, that is, the location of the file (such as the row number ). Therefore, in fact, queue is only a message index. Is it necessary to persist the queue? It can be persistent. After all, the restoration time of queue can be shortened when the broker is restarted. Does it need to be synchronized and persistent with persistent messages? Obviously, this is not required. We can asynchronously and periodically persist each queue, and then restore the queue from the persistent part first, then, the remaining parts are supplemented by persistent messages, so that the parts of the queue that are slow due to asynchronous persistence can be flushed. Therefore, after the above analysis, all messages are stored in nosql, and all the queue is in the memory. How is the message persistent? I think the best way is to have a global sequence number for each message. Once the message is written into nosql, The Global sequence number of the message is determined, then, when updating the corresponding queue information, we pass the global sequence number of the message to the queue, in this way, the queue can establish a ing relationship between the local sequence number of the message and the global sequence number of the message. The related code is as follows: copy the public MessageStoreResult StoreMessage (Message message, int queueId) {var queues = GetQueues (message. topic); var queueCount = queues. count; if (queueId> = queueCount | queueId <0) {throw new InvalidQueueIdException (message. topic, queueCount, queueId);} var queue = queues [queueId]; var queueOffset = queue. incrementCurrentOffset (); var storeResult = _ messageStore. storeMessage (message, Queue. QueueId, queueOffset); queue. SetMessageOffset (queueOffset, storeResult. MessageOffset); return storeResult;} There is nothing better to explain than the code. The idea of the Code on is to receive a message object and a queueId. queueId indicates the number of queue of the current message. Then, the internal logic is to first obtain all the queue of the topic of the message. Because the queue and topic are both in memory, there is no performance problem here. Then, check whether the passed queueId is valid. If it is valid, locate the queue and Add 1 to the internal sequence number of the queue through the IncrementCurrentOffset method. Then, the message is persisted and the queueId and queueOffset are persisted together during persistence, returns the global serial number of a message. MessageStore stores the message content, queueId, queueOffset, and the global sequence number of the message as a whole in nosql. The key is the global sequence number of the message, value is the whole (serialized as binary) as described above ). Then, call the SetMessageOffset method of queue to establish a ing between the global offset of queueOffset and message. Finally, a result is returned. MessageStore. storeMessage memory implementation is roughly as follows: copy the code public MessageStoreResult StoreMessage (Message message, int queueId, long queueOffset) {var offset = GetNextOffset (); _ queueCurrentOffsetDict [offset] = new QueueMessage (message. topic, message. body, offset, queueId, queueOffset, DateTime. now); return new MessageStoreResult (offset, queueId, queueOffset);} copy the code GetNextOffset to obtain the next global message serial number. QueueMessage is the "whole" mentioned above ", Because it is implemented in memory, a ConcurrentDictionary is used to save the queueMessage object. If you use nosql to implement messageStore, you need to write nosql here. The key is the global serial number of the message, and the value is the binary serialized data of queueMessage. Through the above analysis, we can know that we will keep the message's global serial number + queueId + queueOffset together as a record. In this way, there are two very good features: 1) the message persistence and the persistent atomic transaction of the message in the queue; 2) we can always restore all queue information based on these persistent queueMessage, because queueMessage contains the location information of messages and messages in the queue. Based on such message storage, when a consumer wants to consume a message at a certain position, we can first find the queue through queueId, and then obtain the global offset of the message through the message in queueOffset (transmitted by the consumer, then, the global offset is used as the key to get the message from nosql. In fact, the current equeue pulls messages in batches, that is, a socket request pulls a batch of messages instead of a message. The default value is 32 messages. In this way, the consumer can get more messages with fewer network requests, which can speed up message consumption. 3. Details of message routing when the producer sends a message how many queue does the current topic have when the producer sends the message? Do I have to go to the broker to check every time a message is sent? Obviously, the message sending performance will not go up. What should we do? It is asynchronous. The producer can send requests to the broker regularly to obtain the number of queue under the topic and save the requests. In this way, each time the producer sends a message, it just needs to be taken from the local cache. This cache makes sense because the number of topics on the broker does not change. There is another question: where does the queue come from when the current producer sends a message to a topic for the first time? Because the scheduled thread does not know the number of queue under the topic to be retrieved from the broker, because there is no topic in the producer end, because no message has been sent yet. It is necessary to judge that if the current topic does not have the count information of the queue, the queue count information will be obtained directly from the broker. Then cache the message and send the current message. Then, when sending the message for the second time, because the message already exists in the cache, you do not have to get it from the broker, in addition, the scheduled thread will automatically update the count of the queue under the topic. Well, if the producer has the queue count of the topic, the framework can transmit the queueCount of the topic to the user when sending the message, then the user can route the message to the queue according to their own needs. 4. how to Implement consumer load balancing means how to enable consumers in the same consumer group to consume queue in the same topic on average when the consumer cluster consumes. Therefore, this Server Load balancer is essentially a process of evenly assigning queue to consumer. How can this problem be achieved? According to the above definition of Server Load balancer, we only need to determine the consumer group and topic for Server Load balancer, and then get all the consumer under the consumer group and all the queue under the topic; then, for the current consumer, you can calculate the queue to which the current consumer should be allocated. We can use the following function to obtain the queue to which the current consumer should be allocated. Copy the public class metadata: IAllocateMessageQueueStrategy {public IEnumerable <MessageQueue> Allocate (string currentConsumerId, IList <MessageQueue> totalMessageQueues, IList <string> totalConsumerIds) {var result = new List <MessageQueue> (); if (! TotalConsumerIds. contains (currentConsumerId) {return result;} var index = totalConsumerIds. indexOf (currentConsumerId); var totalMessageQueueCount = totalMessageQueues. count; var totalConsumerCount = totalConsumerIds. count; var mod = totalMessageQueues. count () % totalConsumerCount; var size = mod> 0 & index <mod? TotalMessageQueueCount/totalConsumerCount + 1: totalMessageQueueCount/totalConsumerCount; var averageSize = totalMessageQueueCount <= totalConsumerCount? 1: size; var startIndex = (mod> 0 & index <mod )? Index * averageSize: index * averageSize + mod; var range = Math. min (averageSize, totalMessageQueueCount-startIndex); for (var I = 0; I <range; I ++) {result. add (totalMessageQueues [(startIndex + I) % totalMessageQueueCount]);} return result ;}} the implementation in the copy code function is not analyzed much. The purpose of this function is to return the queue allocated to the current consumer based on the given input. The principle of distribution is average distribution. Well, with this function, we can easily achieve load balancing. We can open a scheduled job for each running consumer. The job performs Load Balancing once at a time, that is, executing the above function, obtain the latest queue bound to the current consumer. Because each consumer has a groupName attribute to indicate which group the current consumer belongs. Therefore, we can obtain all the consumers in the current group from the broker during Server Load balancer. On the other hand, because each consumer knows which topics it subscribes to, it has topic information, all the queue information under the topic can be obtained. With these two types of information, each consumer can perform load balancing on its own. First, let's look at the following code: _ scheduleService. scheduleTask (Rebalance, Setting. rebalanceInterval, Setting. rebalanceInterval); _ scheduleService. scheduleTask (UpdateAllTopicQueues, Setting. updateTopicQueueCountInterval, Setting. updateTopicQueueCountInterval); _ scheduleService. scheduleTask (SendHeartbeat, Setting. heartbeatBrokerInterval, Setting. heartbeatBrokerInterval); each consumer starts three scheduled tasks. The first task indicates that the Server Load balancer is performed on a regular basis. The second task table Indicates that the queueCount information of all topics subscribed by the current consumer is updated regularly, and the latest queueCount information is saved locally. The third task indicates that the current consumer sends a heartbeat to the broker periodically, in this way, the broker can know whether a consumer is still alive through heartbeat, and maintain all consumer information on the broker. Once a new consumer is added or finds that the heartbeat consumer is not sent in time, a new or dead consumer is considered. Because the broker maintains all the consumer information, it can provide the query service, for example, query the consumer under a consumer group. With these three scheduled tasks, you can achieve Load Balancing for consumers. Let's take a look at the Rebalance method first: copy the code private void Rebalance () {foreach (var subscriptionTopic in _ subscriptionTopics) {try {RebalanceClustering (subscriptionTopic);} catch (Exception ex. error (string. format ("[{0}]: rebalanceClustering for topic [{1}] has exception", Id, subscriptionTopic), ex );}}}