NET implementation of DDD, CQRS and microservices architectures

Wetext Project: A demo case of the DDD, CQRS and MicroServices architecture based on. NET implementation

Recently, I learned a little bit about the microservices architecture (Microservice Architecture,msa) for job needs. I have spent two weeks in my spare time, based on my understanding of micro-service architecture from scratch. NET creates an application case for demonstrating the microservices architecture, and combines the domain-driven design (DDD) and the command-query separation of Duties (CQRS) architecture patterns to explore some practical aspects of the event-driven microservices system architecture. Now will own thinking and harvest of the written, share to everyone.

Micro-Service Architecture

Before introducing the source code, I would like to talk about the micro-service architecture, although there are many discussions on the micro-service architecture on the Internet, but I think it is necessary to say a few more here. Martin Fowler, a master figure, mentions in his personal home page about microservices that microservices do not have a very clear definition. In fact, there are many kinds of implementations of distributed systems that can be seen (or barely seen) as being oriented toward microservices architectures. Personally, I think the microservices architecture should meet several characteristics:

• The entire system is divided into multiple applications (Monolithic Architecture, or single architectures) with relatively independent business functions (known as "microservices"), each of which typically follows a standard layered architecture style or event-driven architecture style. Be able to handle the domain logic of your own, use a local database for data storage, and provide a relatively independent API interface or user interface to the upper layer. Each microservices can also use infrastructure layer facilities such as caching, logging, but if they are common to other microservices, the infrastructure layer facility needs to meet the following third feature
• Each microservices can communicate using the following methods (see: http://howtocookmicroservices.com/communication/)
  • Synchronization: The most common is based on RESTful style API, can also be cross-platform, cross-language Apache Thrift IDL
  • Async: Use lightweight message communication mechanisms such as RABBITMQ, Redis, etc.
• The entire system is "cloud-friendly" (cloud-friendly). The so-called "cloud-friendly" means:
  • For each microservices, you should avoid the possibility of a single point of failure. For example, for a micro-service A in a system, you need at least two running instances (or more), and either by API Gateway (API gateways) or load balancer (load Balancer) The service request from the client is assigned to a running instance of any a, according to certain rules (such as the health of the running instances of each a) to complete processing
  • For each microservices, administrators can adjust the deployment of these applications based on specific real-time specifications. For example, the query service load of a shopping site is significantly higher than the order management Service, so the administrator can increase the amount of deployment of the query service (such as deploying 3 instances of the query service) and reduce the amount of deployment of the Order Management Service, depending on the actual situation. The benefit of this is obvious when compared to the monolithic architecture of the system as a whole, it can make full use of the cloud server resources, so that each micro-service can run in a reasonable resource configuration state, reduce the waste of resources
  • The Public infrastructure Layer service facility should also meet the criteria for avoiding single points of failure, such as the need for database services to configure replication/clustering, and Message Queuing to use similar fault tolerance policies
  • Based on the "cloud-friendly" requirements, a large number of deployment and operational issues are derived, such as the configuration patterns of the microservices themselves (multiple instances per machine configuration). Or is each machine configured with an instance? Or do you configure one instance per virtual machine? Or is the instance configured in a container like Docker? How can Message Queuing be configured to support multiple instances of the same microservices without repeating the same message? How does restful communication allow clients to find dynamically changing API addresses? Wait a minute. I think that these questions do not have a specific answer, or need to be judged according to the actual situation

Compared with the traditional integrated architecture system, the MicroServices architecture system has the following advantages:

• Each micro-service is relatively small, making it easier to develop and debug
• Each micro-service is relatively independent, so that not only can developers focus on a business processing part, but also for each microservices own characteristics, with different technology implementations (such as partial microservices using C # Implementation, some use Java or Python, etc.)
• This independence allows MicroServices to perform well in fault-tolerant isolation: for example, a crash of a microservices problem does not cause the entire system to be unavailable, which is in accordance with base (basically Available, soft-state, eventually Consistency) Philosophy
• Because of their relative independence, microservices architectures are designed to be more easily deployed in a cloud environment
• The independence of MicroServices also provides a good support for agile development. For example, each service can be independently developed separately, and the project team can balance development and testing resources based on the member's own technical expertise

Of course, it also has some shortcomings:

• Developers need to address the complexities associated with distributed architectures. For example, if the MicroServices use asynchronous message communication mechanism to communicate, then it is necessary to follow the development pattern introduced by this kind of message mechanism (creating message Processor messages Handler, forwarding messages, etc.). In addition, this architecture provides many inconvenient factors for testing, such as when some test cases (testing Cases) need to cover the business of multiple microservices, it is often more complicated to focus on the execution results of weakly consistent distributed transactions. Further, such testing needs to be coordinated by multiple teams in order to be successful, and when teams are distributed across regions of the globe, coordination becomes even more complex and difficult
• It is not easy to deploy, install, and manage microservices-based systems in a production environment. This requires the client side has a strong technical background and the ability to solve problems. Of course, a better way is to deliver services directly to consumers in a SaaS way.
• More resource consumption. For isolation and fault tolerance, microservices are likely to be deployed as N instances, each running in a separate virtual system. Assuming that the deployment strategy improperly results in a certain waste of system resources, this waste can also be expanded n times

About the content of the micro-service architecture for the time being written so much, micro-service architecture is now relatively hot, we can also directly online access to relevant information, English than better friends suggest that the English language website to search for learning, there are many essence of the article and wonderful discussion. The architecture itself is the benevolent see, different people have different understanding, produce different views, some ideas may be more appropriate in some scenarios, but the other scene reflects its weakness. But anyway, I want to say that no matter what architecture to choose, it always has advantages and disadvantages, the difficulty of architecture design is how to choose the most appropriate mode, method, technology to complete a set of system development solutions. In many cases, the entire application system is more likely to be an "ecosystem" that incorporates multiple architectural styles in a variety of technologies. For the project you are developing now, it may be most appropriate to use the classic three-tier architecture.

Wetext Project

There are theories that need to be practiced. To do this, I spent two weeks in my spare time, using Visual Studio 2015 to develop a case project: Wetext. The business of this case project is still very simple: the user can register, login, login can modify the personal information, and then you can create some of their own text (that is, with the title and text content of small notes), can also send friends to apply to other users, and other people accept the invitation, You can share your text with each other. By the end of my writing, the text share has not yet been completed, but the rest of the business has basically gone through, and there may be many bugs.

See here, you will certainly have to spit groove, so simple system still need to spend two weeks, make such a big movement, there are so many bugs, incredibly still did not finish! Yes, it's not perfect at the moment, why? Because of the complexity of the architecture, I am thinking while design side coding, perhaps using CQRS's micro-service architecture is not suitable for such application system, even DDD may not be useful. To use this architectural style on this project, to be honest, I just want to practice. So far, the project has the following shortcomings, but also to your readers to endure. Of course, it's open source (Apache 2.0 License), and you feel like there's no place to have fun. Participate in discussions and contributions, and submit a pull request to me.

• The query part of CQRS uses the relational database, the database access plane does not use ORM, only realizes the table data Gateway mode, but the implementation of table Data gateway is a single-table structure, A cross-table query cannot complete the JOIN operation: An interested friend can implement another ORM-based query mechanism based on an existing Wetext project
• Although the Web Application home page claims to have the event Sourcing, but in fact I do not record any events in the event store, but the final state of the aggregation is saved in the event store (for the sake of time, otherwise one months is not necessarily finished, Time and energy can't afford it. CQRS No event Sourcing,oh my god! But don't be surprised, CQRS do not have to use the event Sourcing: Interested friends can be based on the existing Wetext project to implement the function of event Sourcing, but don't forget to also take the snapshot, this is very important! You can also use the Mature event store framework on Wetext to perform this part of the function. Come to the conclusion, don't forget to share it.
• The command part of CQRS is encapsulated by the RESTful API. Because the command execution is asynchronous (only final consistency is guaranteed), and the RESTful API is synchronous, the RESTful API cannot return the final result of the command execution. I'm thinking about the need to introduce an actor-model-based scenario such as Akka to solve this problem, but it doesn't have to be effective. Still looking for solutions. Interested friends can continue to consider this issue in depth
• The exception handling section is relatively weak: this Part I will continue to strengthen
• The front-end Interface (Wetext.web project) is relatively ugly, there are some drawbacks, is simply the use of ASP. NET MVC 5 combined with Bootstrap, did not use Typescript+angularjs, React or even jquery is doing some high-end user experience, basically satisfying the support of the backend business. Interested friends can throw away the Wetext.web project and develop their own front-end interface using only the services provided by Wetext
• It is not yet fully verified that the deployment in the cloud is feasible, theoretically feasible, but not fully validated, and I have a conclusion.

Overall architecture

First, let's look at the whole structure of the Wetext project from the overall architecture perspective and the various components it contains.

, the blue section represents concepts related to the domain, such as aggregations, statutes, events, sagas, warehousing, and so on; The yellow section represents MicroServices, and currently has accounts, texting, and social three microservices; the gray section represents infrastructure layer facilities, including Owin-based web API host program, message queue, Event store, database, and so on; a light pink color block represents a service host process.

• The client program sends a command request to the server via the RESTful API (Web API)
• The server forwards the request via API Gateway or load balancer to the appropriate MicroServices instance (API Gateway and load balancer are not reflected in, that's another thing that I'll discuss in the future)
• The WEB API controller converts the request to CQRS command and dispatches it to the command Queue
• Command handler get command message, Access domain through repository (this process will involve snapshot), execute commands operation
• Repository when you save an aggregation, the events generated by the operation are stored in the event store (this process involves snapshot), and the domain events are distributed to the Message Queuing event queue
• Event handler performs a message-related operation after acquiring a message, triggering a Saga state transition in the event handler, which, after the saga state changes, produces a state change domain event that event Handler also triggers another command to occur (in theory, it should trigger the command directly in a saga, but the saga itself should also be the aggregation root, so it's obviously unreasonable to have the command distributed directly by the saga, which is discussed later)
• Event handler updates the query Database (that is, the steps in normalize) as needed
• The client's query request is accessed directly through the RESTful API (Web API) via the table Data Gateway, which directly completes

For service host, it provides a hosting environment for three service instances at a time. In fact, Wetext's design allows service host to host only one or a few instances of it, and multiple service hosts can be deployed on multiple different physical machines, such as:

So, throughout the environment, we have a accounts service instance, two texting service instances, and two social service instances. At least a single point of failure and server resource balance provide a solution, of course, also brings a lot of problems. Like what:

• How can I configure the routing of the API gateway so that client requests can be distributed to the corresponding microservices instances on a specified policy?
• For microservices with multiple instances, how does the Pub/sub-based message subscription mechanism avoid repetitive processing of events or commands?

I will discuss these issues in a follow-up article.

In addition, you may think that infrastructure layer facilities have a single point of failure, such as RABBITMQ or databases. In fact, these mature products have their own solutions, such as database cluster. Or simply use the PAAs services provided by AWS or Azure (Message Queuing, storage, and so on). Therefore, it is not difficult to solve this problem.

Begin

In order to better understand the architecture of the Wetext project and the technologies used, it is recommended that you understand the following in advance:

• Domain driven Design (DDD)
• Command query separation of Duties (CQRS)
• Micro-Service Architecture (MSA)
• Message communication mode (Messaging Patterns): can refer to the RABBITMQ official learning materials: http://www.rabbitmq.com/getstarted.html

Next, the important thing, forget it, say it again, use Git to clone the project code to Local:

1	git clone https://github.com/daxnet/we-text.git

You can then open the WeText.sln file directly using Visual Studio 2015. After opening the code, do not rush to run, let us first look at the project structure.

• Services directory: Projects with three microservices: accouts (user account MicroServices), social (social microservices), and texting (small note micro-service)
• Wetext.common Project: The base library that contains the entire solution
• wetext.domain Project: Domain model and command, event definition
• wetext.domainrepositories Project: Specific implementation of domain warehousing (MongoDB implementation)
• WeText.Messaging.RabbitMq Project: Implementation of message system based on RABBITMQ
• WeText.Querying.MySqlClient Project: mysql-based table Data Gateway implementation to provide CRU operations on MySQL database (delete is not supported temporarily)
• WeText.Querying.PostgreSQL Project: PostgreSQL-based table Data Gateway implementation to provide CRU operations to the PostgreSQL database (delete is not supported at this stage)
• Wetext.service Project: The host program of the MicroServices, after startup is a console server program (the runtime starts this project first)
• wetext.tests Project: NUnit-based unit test project, please ignore directly
• wetext.web Project: Front-end user interface project, start the project after the Wetext.service project starts
• In addition, in the We-text root directory, there is also a scripts subdirectory containing the initialization script for MySQL and PostgreSQL database (when writing this article, PostgreSQL script is not joined, then add), in the subsequent installation steps will be used

External dependencies (External Dependencies)

First, Wetext relies only on the relevant libraries required by some infrastructure layer facilities, including:

• Mysql.data
• Rabbitmq.client
• Mongodb
• Npgsql
• AUTOFAC related
• Owin related
• Newtonsoft Json
• Log4net
• Libraries related to ASP.

In addition to this, there is no application-layer development framework and code base, and all of the code is original and contained throughout the Wetext solution.

Second, the service-side infrastructure layer fully selects projects and products such as Owin, MySQL, RabbitMQ, PostgreSQL, etc. that can be cross-platform, so that the entire Wetext server can be fully deployed in a Linux environment (which is, in fact, a part of what I want to practice, Verify that the. NET Server program based on Mono is performing well on the Linux system. There is no use of SQL Server, the Entity framework, which are currently more suitable for products running on the Windows platform.

Installation and operation

First, for convenience, it is highly recommended that all services and programs be installed on the same machine. To prepare the system environment, follow these steps:

1. Download the source code (refer to the git command above)
2. To download and install RABBITMQ, the installation process uses the default configuration (including service ports, and so on), for RABBITMQ installation, see: https://www.rabbitmq.com/download.html
3. To download and install MongoDB, the installation process uses the default configuration (including service ports, etc.), for MONGODB installation see: https://docs.mongodb.org/manual/installation/, If you need to change the Wetext mongodb configuration after installation, please go to the MongoSetting.cs file under the Wetext.domainrepositories project (when writing this article or hard code in the code, will move to App. config in the future)
4. Download and install the MySQL Community Edition (including server and client), the installation process using the default configuration, the root password please use [email protected]. For installation of MySQL, see: http://dev.mysql.com/doc/refman/5.7/en/ Installing.html, if you need to change the MySQL configuration of wetext after installation, please modify the App. Config file of the Wetext.service project directly
5. If you plan to use PostgreSQL as a query database, you only need to install PostgreSQL, no need to install MySQL. The installation process also uses the default configuration
6. Initialize the corresponding database using the SQL script under the scripts directory under the We-text project folder, the current PostgreSQL script is not added, and then add

Once the environment is ready, you can try to start the project.

To start and debug a project in a Windows system

1. To open a WeText.sln file by using Visual Studio 2015
2. To start the Wetext.service project, you should see the following screen:
   
3. To start the Wetext.web project, you should see the following screen:
   
4. Try clicking Register to register your account and log in

Compiling and starting the server program in Linux

Note: I haven't had time to test a server deployed on Linux using the Wetext.web site, only trying to compile and start the server program in a Linux environment. The Web Site program (wetext.web) itself is not intended to run in a Linux environment and can be tried later.

2. To install mono, follow this page to complete the installation of Mono: http://www.mono-project.com/docs/compiling-mono/linux/. Recommended to install directly from the release package, you can download the latest version of Mono to http://download.mono-project.com/sources/mono/. Wetext requires support for the. NET Framework 4.6.1 and C # 6.0
4. It is also necessary to prepare the system environment based on the above steps, including RABBITMQ, MongoDB, and query database installation and initialization.
6. Download the source code using the git command above
8. Since there are still some problems with the support of NuGet under Mono, some of the package cannot be downloaded, it is recommended that you compile Wetext under Windows and upload the downloaded packages directory to Linux we-text\ SRC Directory
10. Enter the we-text\src directory and use this command to complete the compilation:xbuild/p:targetframeworkversion=v4.6.1/p:configuration=serverdebug WeText.sln. There may be some warnings for compilation, please ignore them directly
12. Go to the we-text\bin directory and execute the ./wetext.service.exe command, you should see the following screen:
    
14. To try to access services on Linux from the Wetext.web site, temporarily look for the http://localhost:9023/string in the Wetext.web project and replace localhost with the service URL of the Linux host.

Summarize

In this paper, the micro-service architecture is introduced briefly, and a demonstration case of Wetext, the DDD, CQRS and microservices architecture based on the. NET implementation is introduced from the aspects of the overall architecture, the use of the code base, the environment preparation and the compilation deployment. Friends interested in micro-service Welcome to try this case source code, and welcome to participate in more in-depth discussion. Wetext is still not very mature, I will gradually improve the case, but also in this process to share their experience, welcome attention.

Category: Domain driven design, microservices

NET implementation of DDD, CQRS and microservices architectures