Enode Framework Conference Case Analysis series--how event traceability deals with refactoring issues

Objective

This article may be more difficult to understand for most of the friends who do not know Enode, this article focuses on ideas, not on the results. I write articles, always hope to be able to put their own thinking process as far as possible to express, can let you know every design behind the thinking of things. I think that the result of any design may look very tall, a picture can be, but the thinking behind, is more valuable things.

This article wants to write about how Enode handles the problem of model refactoring caused by changes in business requirements. DDD is the solution to complex business problems because DDD is a model-driven software design approach. The domain model is used to capture the business requirements, and the domain model satisfying the demand is abstracted according to the business requirements, and the model can be continuously refined and evolved with the changing business requirements. I think, to sum up, the reconstruction of the domain model is mainly in the following four situations:

1. You only need to modify the business logic without modifying the model structure, such as simply modifying the judgment logic of a business rule;
2. Need to make some small changes to the model structure, such as adding, renaming, deleting a property;
3. Need to be large-scale operation of the model, such as the need to split the original aggregation into two separate aggregations, such as e-commerce system, the original commodity inventory information may be in the commodity aggregation root, but later because of the complexity of inventory management, need to put inventory independent to the new context (inventory context), and independent maintenance;
4. There are also situations where it is possible to downgrade an aggregation root to a sub-entity under another aggregation root, or to upgrade an entity to an independent aggregate root. This situation I personally think that generally do not appear, if it appears, is generally the original domain model design has resulted in large problems, and this situation can generally be solved by adding an independent aggregation root;

Traditional application in the face of the above reconstruction processing method

A traditional application is a four-tier architecture that does not use event Sourcing (ES), such as a classic three-tier architecture or classic DDD, which stores the latest state of all business data through a database. This architecture is handled in the following ways:

1. Only modify the business logic situation: Directly modify the logic, the database does not need to make changes;
2. New, renamed, and deleted model properties: For new attributes, we only need to add fields to the DB table, and then the new fields will use the default values, and for the property renaming, I think we just need to modify the name of the property at the level of the code. The name of the table field in the DB does not need to be modified (if you want to change the field name, it is a large project), for the deletion of the model properties, it is generally only necessary to modify the code, the DB table field does not need to be deleted, or you must delete also have no effect;
3. For an aggregation root split into two aggregate roots: processing is more complex, generally need to pass a new table, data migration, code logic switch several steps;
4. For aggregation root demotion or escalation scenarios: sometimes it is simpler because, although the relationship between aggregations is changed at the level of the code, the structure and relationships of the tables are not changed (generally as long as our database tables are designed for the third paradigm, the database is often stable). In this case, it is simpler to refactor the code directly, but sometimes it is possible to modify or even change the table structure, and this situation needs to be treated like 3.

For 3, 42 scenarios need to do data migration and switching situation, is generally the system needs to change, and generally does not occur frequently. If it does, we need to sit down and think about what to do with the actual situation. Architects are designed to design code refactoring scenarios, data migration scenarios, and data access switching scenarios. If the switchover fails, also support fast rollback; In short, this is a big project. Fortunately, we now have some mature and even automated data migration solutions (reusable) or tools that can help us greatly simplify the rebuilding cost of refactoring.

How schemas are handled using ES (event Sourcing, incident traceability)

Using ES systems, it should be recognized that the problem of refactoring is much more complex than traditional applications. For example, in addition to the most recent state of the aggregate root stored in the database, there is more than one eventstore,eventstore that stores all the historical events generated by each aggregate root. In addition, because it is an EDA (event-driven architecture), the data is not only in db, but also a subset of messages (command or event) are still in the queue and are not processed. So we have to deal with the messages that remain in the queue. Therefore, in the application of ES architecture, in addition to migrating the data of the latest state of the aggregate root in the CQRS read Library, there are additional two things to consider when migrating or switching data: 1) The migration of events in Eventstore; 2) processing of messages not processed in the queue; So, the application of ES architecture, When dealing with such big changes, the solution may be much more complex than the traditional. This is what I want to write the purpose of this article, I hope that through a fictitious example (in real life may also occur), to analyze the ES architecture, how we deal with this situation, we hope to be able to use the ES architecture development applications of friends to provide some reference.

Then in the case of the above 4 refactoring, 1 is relatively simple, but only to change the logic of the place where the event occurs, 2 is simpler, only need to modify the corresponding event, add or remove attributes in the event class, anyway JSON serialization and deserialization are compatible; 3, 42 cases are more complex, There are more things to consider when migrating our data than traditional architectures. I already mentioned it above.

Example of splitting an aggregate root

As described in the previous articles, we know that in the Conference case, the Conferencemanagement context (meeting management context) is responsible for managing the basic information of the meeting and the inventory information for all seats in the meeting. But assuming we are in the future with the complexity of inventory management, we want to split the logic of inventory management into a separate context, the meeting seating inventory management context (bounded context, BC). In this case, it is equivalent to splitting the original conference aggregation root into two aggregate roots, the original conference aggregation root is only responsible for maintaining the basic information of the meeting, no longer maintain the seat inventory information The inventory information is then given to a separate aggregation root (conferenceseatinventory) maintenance. A conference aggregate root will have a unique conferenceseatinventory aggregation root counterpart, but the ID of the Conferenceseatinventory aggregation root can be independent, The ID of the conference aggregation root can then be used as a property of the Conferenceseatinventory aggregation root. Just like the payment aggregation root there is a OrderID like, hehe. With this split, the context of our meeting management needs to focus on the basic information of the Conference itself, including the design of the UI. Then there is a dedicated inventory management system (including UI) that is responsible for managing the inventory information for the meeting seat.

The scene of this split aggregation root is what I imagined, and it may not happen, and it may happen, depending on how our business develops. Just like Ali's inventory center may not be independent at the earliest time, but with the commodity center together. But later with the attention to inventory management or inventory management (especially inventory reduction) of their own business complex, we put inventory management independent.

Model refactoring

Before we do the data migration, we need to make the new inventory management context and launch it. But at this point, there is no command coming from this context.

Ideas for migrating and switching data

An application developed using Enode is a cqrs+eda+es architecture. Then, as we know, there are three types of data that we want to migrate under this architecture:

1. Events in the Eventstore;
2. CQRS read the data in the library;
3. Messages that have not been consumed in the distributed queue (equeue): commands, events;

My train of thought is:

2. The first is to minimize the extent of the data migration impact. Therefore, we can first isolate all current and conference aggregation root of the inventory change related command and event messages to separate queue. This is simple because Enode currently uses a distributed message queue that is Equeue, and Equeue sends or subscribes to messages based on topic. When we want to send a message, we need to tell Equeue which queue is currently being sent to which topic, and when we want to subscribe to the message, we tell equeue which topic to subscribe to. Therefore, we can configure a separate topic for conference aggregation root and inventory change related commands, events (such as called Changeconferenceinventorycommandtopic, Conferenceinventorychangedeventtopic). In this way, we only need to start the data migration from these two separate topic.
3. The
4. adds a separate event Handler, which is responsible for subscribing to events related to inventory changes in the Conference aggregation root, that is, subscribing to events topic Conferenceinventorychangedeventtopic. Its processing logic is based on events, creating and persisting new and new aggregation roots (conferenceseatinventory) corresponding to the inventory modification event to Eventstore. But the situation here is rather complicated and I need to expand it in detail. For example, we currently receive an event (the event version number is 10), and then find that the corresponding conference aggregation root for this event does not have any corresponding conferenceseatinventory corresponding to the event, because it is the first synchronization. At this point we need to take all previous historical events of the Conference aggregation root (all events with version number 1 to 10) out of the Eventstore, and then convert the events related to inventory changes to the corresponding new events, Then save all of these new events to Eventstore. Here's a question: how do we know if the previous events of the current aggregate root are all synchronized? One simple approach is that each time an event comes up, we all check from Eventstore whether the event has been synchronized all the previous events, and if so, it is possible to synchronize the current event directly. If not, then you need to synchronize the previous events first. But this is undoubtedly inefficient. In fact, we can store the version number of each aggregate root event that is currently synchronized in the memory of the current machine. That is, a concurrentdictionary is used to save the version number of synchronized events for each aggregate root. Then, when judging, only need to judge according to this dictionary, this is undoubtedly a lot of efficiency improvement. But why just cache the synchronization progress in the memory of the current machine? Why not use distributed caching? Because all of the command or event messages in Enode are routed based on the aggregate root ID. Messages with the same aggregate root ID are always routed to the same queue. According to Equeue's architecture, the same queue consumer is always the same. So, we can do that.
5. The
6. event handler above only deals with new events generated by the Conference aggregation root. But what about the aggregation roots that have previously produced inventory modifications but have not been produced recently? We need to develop an event sync task that scans all of the conference aggregation roots in Eventstore for all and inventory modification related events, and translates these events into inventory modified events for the new aggregation root. It is easy to see that this task and the event handler above may cause concurrency conflicts. This concurrency conflict is expected, and we can solve it by technical means. For example, we can do optimistic concurrency control by using the version number of the event that the conference aggregates the root of the synchronization. Concurrentexception can be thrown when the same version of an event is persisted in the target's eventstore with the same target aggregation root (conferenceseatinventory). Then in our code, when we encounter this concurrency exception, we only need to query the latest version of the event to synchronize, and then try to synchronize the event after this version of the event. Here's a question to mention, which is that the events table of the Eventstore database may hold all of the event of the aggregate root, so that the data in the events table is very much. Therefore, when designing the events table of Eventstore, we should try to do a vertical split of the business plane. That is, the events isolated from different types of aggregation roots, such as conference aggregation root events, are placed in the Conferenceevents table, and all events payment The aggregation root are placed in the Paymentevents table. As a result, we don't have as many records in our event table. But even an aggregate root of all events can be quite a lot. Then we need to split the horizontal, that is, all the events of the same aggregate root are then divided into a separate table. This is an off-topic, this is just a mention. Our purpose here is to focus on synchronizing all the events in the existing conferenceevents table, converting these events into events of the new aggregate root, and persisting to the new event table.
8. after 2, 32 steps, we ensured that all the historical events in the last Conferenceevents table and the ever-increasing increment events were automatically synchronized to the new event table (conferenceseatinventoryevents). That is, target 1 of our target, and then we start to think about how to generate the CQRS of the inventory aggregation root for the synchronization of the Read library data. It is simple to update the read library data of the inventory aggregation root, just to design another independent event scan task, which is responsible for scanning the conferenceseatinventoryevents table and scanning for any subsequent events that are not currently scanned. and update them to read libraries one by one. We need to keep track of which record is currently scanned in memory (and we need to save the scan progress on a regular basis, such as writing to a file, to the point where the scan task stops unexpectedly or the machine shuts down unexpectedly, so that we can load the last location of the last scan from the file and then continue scanning from that location) , and then continue scanning every time you go from one record to the next. You can scan up to 1000 records at a time and scan at intervals of 10s. These we can configure ourselves. A friend who has seen the design of the Enode event table should know that the Enode Design event table has a self-incrementing sequence field that is designed to be used for incremental scanning or synchronization of events, hehe. Without the sequence field, we would not be able to know the global order in which the records were added to the table. Well, with this scanning task, we can finally make sure that the read Library of the Conferenceseatinventory aggregation root is updated in an asynchronous way.
10. Next we think about how to handle commands and events that have not yet been consumed in the queue. Why should we consider this issue? Because our requirements are to be able to do as far as possible without downtime release. That is, the migration of this data and the process of switching is as transparent as possible to the user. To do this, we must ensure that the inventory-related events of the new and old aggregate roots and the state of the read library must be exactly the same. Otherwise, when switching to the new aggregate root, because there are still some commands or events in the old aggregation root that have not yet been processed, it will cause us to switch to the new aggregate root and use the old (stale) data, which is not the case. So how do you solve this problem? Because of the first step ahead, we have separated the relevant topic, there are: Changeconferenceinventorycommandtopic, Conferenceinventorychangedeventtopic of these two topic. Then we just have to make sure that all the news under these two topic is consumed. But as long as there is new news coming into the two topic queue, there is no such time. Fortunately equeue at the beginning of the design, considering this situation, equeue support to disable a queue, disabled, the message sender will not be allowed to send messages to this queue, but this queue of messages can also be allowed to be consumed. So, when we want to start switching, we can disable all the queue of the two topic first. Then make sure that no new messages will be sent successfully. Then just wait a few seconds and wait for all the messages under these two topic to be consumed (you can see if the messages are consumed by looking at the Equeue on the Management console queue), and the above-mentioned 2,3,4 three steps also handle all events, That means that all inventory-related events and read-Library data for the new and old aggregate roots are fully consistent. Then we can switch it on.
12. How do you switch it? You only need to switch the source of the sending command. is to send the original to conference aggregation root of the modified Inventory command, now sent to the Conferenceseatinventory aggregation root of the modified inventory command. We can deploy the new code to all the servers that might be sending the relevant commands and then republish it. Of course, this release process may take several minutes. So, as you can see, the whole process of switching may take a few minutes or so.

Summarize

The above thinking I thought for several days, really cost me a lot of brain cells. The traditional DB-oriented data migration scheme typically has full-volume data migration and incremental data migration, as well as the final switching operation.

While Enode is dealing with this situation, because the entire architecture is the EDA architecture, we can take advantage of the EDA architecture in data migration (when events occur, the outside can be notified, and we can add additional event handler at any time to do additional things). The above steps of thinking, is the use of this idea, so that the program can be done without intrusion on the premise of the completion of data migration and system switching.

Another point, do not know that we found no, the above data switch is actually not as complex as imagined. Why is it? Because of the invariance of the event. The records in our event table will never change, unlike the traditional DB data migration scheme, where every row of records in the table is likely to change or be deleted, which poses a major obstacle to data migration. The architecture of ES, because of the invariance of event records, also makes it easier for us to migrate data. Moreover, we also make reasonable use of the features of the CQRS architecture, that is, the update of the read library is entirely based on the event table. So, as long as we ensure that the event tables are absolutely fully synchronized, it is always possible to ensure that the last read library can be completed synchronously. This is also very cool.

Finally, the question I'm thinking is, is it possible to shorten the switching process to a few seconds? Oh. Should be possible, that is, we can pre-change all the code to be modified first, and then through a configuration item to determine which code is currently running, which command to send. Then when we need to switch, we just need to modify the configuration item. This can be done with tools such as zookeeper. In this way, the entire switching process takes just a few seconds and does not have a significant impact on the user. And if you find a problem with switching past, you can switch back at any time. To achieve the rollback of the scheme.

Finally, you can think about this data migration and switching scenarios, where can be reused?

Enode Framework Conference Case Analysis series--how event traceability deals with refactoring issues