Step by step in enode framework

Preface

Today is a happy day and a weekend. You can write and write articles with peace of mind. After about three years of DDD theory accumulation, and the development and project practical experience of the event sourcing framework of the first version at the beginning of last year, we will use our spare time for design and development in the first half of this year, my enode framework can finally meet with you.

It has been many years since Eric Evan proposed the drive design for the DDD field, and many people are learning or practicing DDD. However, I found that there are not many frameworks currently supporting DDD development, at least in China. As far as I know, java and. net platform, which is well known abroad: the axon framework based on the java platform is very active and the author is very diligent. This framework has been used in some practical commercial projects, the calculation is successful. net platform is ncqrs. This framework is very active early in the morning, but it has not developed yet, because almost no one is maintaining it, which is disappointing. There are: banq's jdon framework supports the development of DDD + CQRS + EventSourcing, but it is based on the java platform, so.. net platform is of no practical use ;. net platform, open-source is mainly the apworks framework developed by brother Qingyang in the garden. Brother Qingyang has made great contributions to DDD in China. He has written many articles in the DDD series, and has parallel frameworks and cases. Of course, my focus is on the frameworks of c # And java. There are also many frameworks implemented based on scala and other languages. I will not give them one by one here.

The above frameworks have their own characteristics and advantages, so we will not review them here. If you are interested, let's take a look. What I want to focus on is my enode framework, the characteristics of the framework, and the prerequisites for use.

Introduction to enode framework

1. Framework name: enode
2. Framework features: it provides a DDD-based architecture that implements CQRS + EDA + Event Sourcing + In Memory. It supports load balancing and lightweight application development frameworks.
3. Open Source Address: https://github.com/tangxuehua/enode
4. Nuget package Id: enode

Before using this framework, you must understand or observe the following conventions:

1. A command only allows modifications to one aggregation root or creation of one aggregation root. If this rule is violated, the Framework does not allow modification;
2. If a user operation involves the modification of multiple aggregation root, it needs to be implemented through saga (process manager); embracing final consistency, simply put, the command + domain event is continuously connected to achieve final consistency. If you want to completely know where enode is different, you can take a look at the BankTransferSagaExample in the source code, I believe this will make you understand what I mean by event-driven design;
3. The core programming philosophy of the Framework is asynchronous message processing and final consistency. Therefore, if you want to achieve strong consistency requirements, this framework is not suitable, at least not yet;
4. The framework is not designed for enterprise application development. Traditional enterprise applications generally have low access traffic and require highly consistent transactions. The enode framework is designed for Internet applications, in particular, it provides support for some Internet sites that need to support large access volumes, high performance, scalability, and eventual consistency. I have seen the best practices of scalability: people from eBay experience should know that asynchronous programming and eventual consistency are necessary to implement a scalable Internet application. In addition, if the data volume is large, we generally store data separately, which means that if you want to achieve strong consistency, distributed transactions are required. However, unfortunately, the cost of distributed transactions is too high. Scaling, performance, and response latency are all negatively affected by the cost of Distributed Transaction coordination. As the number of dependent resources and the number of user accesses increase, these indicators will all deteriorate in a geometric level. Availability is also limited because all dependent resources must be in place.
5. Framework positioning: currently, it is positioned at the command end under the CQRS architecture of a single application running on a single machine. To achieve distributed integration between multiple applications on multiple machines, then, we need to further use the ESB to integrate with the higher-level SOA architecture;

Enode framework architecture:

CQRS Architecture

The architecture diagram above is the internal implementation architecture of the enode framework. Of course, the above architecture diagram is not a complete CQRS architecture diagram, but an implementation architecture of command in the CQRS architecture diagram. The complete CQRS architecture diagram is generally as follows:

As we can see, the traditional CQRS architecture diagram usually draws a wide range. There are many implementation solutions for how to implement the command end. The enode framework is only one of the implementations.

Internal implementation of the enode framework

1. First, the client sends the command to the command service. After the command service receives the command, It routes the command queue to which the command should be placed through a command queue router, each command queue is a message queue that stores commands. The message queue is a local queue, but supports message persistence. That is to say, after the command is put into the queue, even if the machine is down, the message will not be lost after the next machine restart. In addition, we can configure multiple command queue as needed. For the sake of illustration, we only drew two;
2. Command queue has a command processor at the exit. command processor is responsible for processing command. However, the command processor does not directly process the command. It directly processes some worker threads in the command processor. Each worker thread continuously extracts the command from the command queue, then, process the command according to the five steps marked in the figure. It can be seen that the worker threads in command processor work in parallel, so we can find that multiple commands are processed at the same time. Why? Because the client sends a command to the command queue quickly, for example, sending 1 million commands per second, that is, the concurrency is 1 W. However, if the command processor only processes the command by a single thread, the speed cannot keep up with the concurrency, so we need to design and support multiple workers to process command at the same time, so the latency will be reduced. We can see from the architecture diagram that, command processor obtains the aggregation root from the memory cache (such as redis that supports distributed cache), which has high performance. Persistence events use MongoDB, because mongoDB has high performance; if we think there is still a bottleneck when the event persists to a single MongoDB server, we can cluster the MongoDB server and then sharding the event to store different events to different MongoBD servers, in this way, the persistence of the event will not become a bottleneck; in this way, the processing performance theory of the entire command processor It can be very high, of course, I have not tested the performance of the cluster, the performance of a single mongodb server, the persistence event, 5 K is not a problem; here is a note, persistence is not a single event, but an event stream, that is, EventStream. Why is event stream because a single aggregation root may generate more than one domain event at a time, but these events are persisted together, so the design idea is to design these events as an event stream, then insert the event stream as a mongodb record to mongodb. The primary key of the event stream in mongodb is the version number of the aggregated root ID + event stream, and these two joint fields are used as the primary key, used to implement optimistic locks. If two event streams target the same aggregation root and have the same version number, a primary key index conflict is reported when being inserted to mongodb, this is the concurrency conflict. The command needs to be automatically retried (the enode framework will help you automatically cancel this automatic retry) to solve this problem;
3. After the worker in command processor processes a command, it will release the generated event to a suitable event queue. Similarly, an internal event queue router will be used to route the specific event queue. How to deal with the event in the event queue? That is, what will event processor do? It is easy to distribute events to all event subscribers, that is, dispatch events to subscribers. What will all these events subscribers do? Generally, we do two kinds of processing: 1) because the CQRS architecture is used, we cannot only persist domain events, we also need to update the CQRS query database through domain events (this kind of event subscriber for updating the query database is generally called denormalizer). Because there is no need to synchronize the update query database, we need to design the event queue; 2) As mentioned above, some operations may affect multiple aggregation root, such as bank transfers, order processing, and so on. These operations are essentially a process. Therefore, our solution is to asynchronously concatenate the entire process by sending command in the event handler of the domain event. Of course, how to implement this process, there are still many questions to discuss. I personally think that a more reliable solution is through process manager, similar to BPM. Many people in foreign countries call it saga. If you are interested in saga or process manager, let's take a look at Microsoft's example: volume manager. As there is too much information, I will write a special article about it.

Review the key technologies used by the enode framework

Based on the architecture diagram of the entire enode framework and the text description above, let's take a look at the key technologies used by the framework mentioned in the previous framework introduction.

1. DDD: refers to the domain model in the architecture diagram. It uses the idea of DDD to analyze and design implementation. The enode framework will provide the basic class aggregation root necessary to implement DDD and support for triggering domain events;
2. CQRS: The entire enode architecture implements the command end in the CQRS architecture, the query end of the cqrs architecture, and the enode framework has no restrictions. We can design it at will;
3. EDA: the idea of the entire programming model is based on the event-driven idea, that is, the State Change of the domain model is based on the response event, and the interaction between the aggregation root, it is not based on transactions, but on event-driven and response;
4. Event Sourcing: Event tracing in Chinese. For more information about Event tracing, see this article. Through event tracing, we don't need ORM to persistently aggregate the root, but only need to persist the domain events. To restore the root, we only need to trace the event of the root;
5. In Memory: all data of the entire domain model is stored In the Memory cache, for example, In the distributed cache redis, And the cache will never be released. In this way, when we want to get the aggregation root, we only need to get it from the memory cache, so it is called in memory;
6. NoSQL: nosql products such as redis and mongodb are used by enode;
7. Server Load balancer support: an application based on the enode framework can easily support Server Load balancer. Because the application itself is stateless, in memory is stored in the global apsaradb for redis distributed cache, independent of the application itself. event store uses MongoDB, which is also global and supports clusters. Therefore, we can deploy any number of applications developed based on the enode framework on different machines, and then perform load balancing, so that our applications can support higher concurrent access.

Framework API usage overview framework Initialization

Public void Initialize () {var connectionString = "mongodb: // localhost/EventDB"; var eventCollection = "Event"; var eventPublishInfoCollection = "EventPublishInfo"; var eventHandleInfoCollection = "EventHandleInfo "; var assemblies = new Assembly [] {Assembly. getExecutingAssembly ()}; Configuration. create (). useTinyObjectContainer (). useLog4Net ("log4net. config "). usedefacommandcommandhandlerprovider (assemblies ). useDefaultAggregateRootTypeProvider (assemblies ). useDefaultAggregateRootInternalHandlerProvider (assemblies ). useDefaultEventHandlerProvider (assemblies) // use MongoDB to support persistence. useDefaultEventCollectionNameProvider (eventCollection ). useDefaultQueueCollectionNameProvider (). useMongoMessageStore (connectionString ). useMongoEventStore (connectionString ). useMongoEventPublishInfoStore (connectionString, eventPublishInfoCollection ). useMongoEventHandleInfoStore (connectionString, eventHandleInfoCollection ). usealldefaprocprocessors (new string [] {"CommandQueue"}, "RetryCommandQueue", new string [] {"EventQueue "}). start ();}

Command Definition

[Serializable]public class ChangeNoteTitle : Command{    public Guid NoteId { get; set; }    public string Title { get; set; }}

Send command to ICommandService

var commandService = ObjectContainer.Resolve<ICommandService>();commandService.Send(new ChangeNoteTitle { NoteId = noteId, Title = "Modified Note" });

Command Handler

public class ChangeNoteTitleCommandHandler : ICommandHandler<ChangeNoteTitle>{    public void Handle(ICommandContext context, ChangeNoteTitle command)    {        context.Get<Note>(command.NoteId).ChangeTitle(command.Title);    }}

Domain Model

[Serializable]public class Note : AggregateRoot<Guid>,    IEventHandler<NoteCreated>,    IEventHandler<NoteTitleChanged>{    public string Title { get; private set; }    public DateTime CreatedTime { get; private set; }    public DateTime UpdatedTime { get; private set; }    public Note() : base() { }    public Note(Guid id, string title) : base(id)    {        var currentTime = DateTime.Now;        RaiseEvent(new NoteCreated(Id, title, currentTime, currentTime));    }    public void ChangeTitle(string title)    {        RaiseEvent(new NoteTitleChanged(Id, title, DateTime.Now));    }    void IEventHandler<NoteCreated>.Handle(NoteCreated evnt)    {        Title = evnt.Title;        CreatedTime = evnt.CreatedTime;        UpdatedTime = evnt.UpdatedTime;    }    void IEventHandler<NoteTitleChanged>.Handle(NoteTitleChanged evnt)    {        Title = evnt.Title;        UpdatedTime = evnt.UpdatedTime;    }}

Domain Event

[Serializable]public class NoteTitleChanged : Event{    public Guid NoteId { get; private set; }    public string Title { get; private set; }    public DateTime UpdatedTime { get; private set; }    public NoteTitleChanged(Guid noteId, string title, DateTime updatedTime)    {        NoteId = noteId;        Title = title;        UpdatedTime = updatedTime;    }}

Event Handler

public class NoteEventHandler :    IEventHandler<NoteCreated>,    IEventHandler<NoteTitleChanged>{    public void Handle(NoteCreated evnt)    {        Console.WriteLine(string.Format("Note created, title：{0}", evnt.Title));    }    public void Handle(NoteTitleChanged evnt)    {        Console.WriteLine(string.Format("Note title changed, title：{0}", evnt.Title));    }}

Key issues to be discussed later

1. Since it is message-driven, how can we ensure that messages are not lost;
2. How to ensure that messages are executed at least once and cannot be repeatedly executed;
3. How to ensure that the message cannot be lost without successful execution, that is, the message queue is required to support transactions;
4. Because it is a multi-thread parallel persistence event and deployed by Server Load balancer on multiple machine clusters, how can we ensure that the order in which domain events are persisted is exactly the same as that in which events are published to event subscribers;
5. In the entire architecture, memory cache based on redis and eventstore Based on mongodb are two key storage points. How can we ensure high throughput and availability;
6. Because events are concurrently persistent, how can we solve concurrency conflicts?
7. How does one retry a command? How does one implement the message Retry Mechanism in a message queue?
8. Since the transaction concept of strong consistency is abandoned and process manager is used to achieve root interaction of aggregation, how can we implement a process manager?

Currently, I think of the above eight important questions. I will discuss the solutions to these problems one by one in the following articles. I think writing an article about this framework should introduce the framework itself, and tell others about the problems encountered during the design and implementation of the framework as well as the solutions. This is the greatest significance for readers to write out the analysis and solution ideas;