Introduction to the Enode framework: A design approach to message queues

In the last article, it briefly introduces the whole realization idea inside the enode frame, and uses the thought of staged event-driven architecture. Through the previous article, we know that there are two types of queues inside Enode: Command queue, event queue; The command sent by the user enters the command queue, and domain model generates domain The event will enter the event queue and wait to be dispatch to all event handlers. This article describes how the two message queues in the Enode framework are designed.

It is helpful to understand the later analysis by pasting the internal implementation architecture diagram of the Enode framework.

What kind of message queues do we need?

Enode is designed to provide DDD+CQRS+EDA based application development within a single process. If our business needs to interact with other systems, it can also be done by interacting with other external systems in event handler, such as broadcasting messages or invoking remote interfaces. Perhaps in the future, Enode will also have built-in capabilities to support remote messaging communications. However, not supporting remote communication does not mean that Enode can only develop stand-alone applications. The Enode framework requires three of the data to be stored:

messages, including command messages and event messages, are currently stored in MongoDB for performance reasons and are persisted because messages in message queues cannot be lost;

The aggregation root, which is serialized and stored in the memory cache, such as Redis or memcached;

An event, an event that is generated by an aggregation root, is stored in a eventstore, such as a mongodb;

Well, through the above analysis, we know that all the data in the Enode framework runtime is stored in the two locations of MongoDB and Redis. Both of these stores are deployed on separate servers, regardless of the Web server. So every Web server running the Enode framework is stateless. Therefore, we can easily cluster the Web server, we can at any time when the user access to increase the number of new Web servers to improve the responsiveness of the system; Of course, when you find that as the Web server increases, it leads to a single MongoDB server or a single Redis server Do not come to the bottleneck, you can also do MongoDB and Redis cluster, or the data to do sharding (of course, these two practices are not very good to do, need to mongodb,redis very familiar with the line), so you can improve the Mongodb,redis throughput.

Well, the above analysis is mainly to illustrate the scope of use of the Enode framework, and it is helpful for us to analyze what kind of message queues we need to discuss.

We now know that we do not need distributed Message Queuing at all, such as the need for MSMQ, RABBITMQ, and other heavyweight mature message queues that support remote messaging. The message queues we need are characterized by:

Memory-based Message Queuing;

Although based on memory, but the message can not be lost, that is, the message to support the persistence;

Message Queuing to be as high as possible;

Message queue, when there is no message, the queue of consumers can not let the CPU idling, CPU idling will directly lead to CPU occupancy 100%, resulting in machine can not work;

To support multiple consumer threads to fetch messages from the queue at the same time, but the same message can only be handled by a single consumer, that is, a message can not be taken away by two consumers, that is, to support concurrent dequeue;

Need a design, implementation message will be processed at least once; When a message is taken by a consumer and then processed, if it is not handled successfully (the consumer knows that there is no deal to be done) or is not at all urgent (such as a power outage at that time), it needs a design that will allow us to consume the message again. ；

Because we're less than 100%, we don't duplicate a message, so all of our message consumers should try to support idempotent operation, that is, repeated operations do not cause side effects; For example, whether the query exists before inserting is a measure to support idempotent; this, the framework will try to provide support for idempotent logic, of course, When you design command handler or event handler, you should also try to consider the question of idempotent. Note: General Command handler do not consider, we mainly want to consider is the event handler. Why, let's discuss it in the next article.