Asynchronous computing architecture for quasi-real-time data processing

Previous Article I have mentioned that we have a requirement: Some data requires quasi-real-time results, but these data cannot be reflected by directly querying business data, most of the data needs to be reflected through a series of complex operations. The effectiveness of such data is a big problem.

The trigger of one business data point actually affects the changes of multiple data indicators. Each indicator data is a specific business logic, if you put these many indicators into a transaction, the execution is obviously unlikely. The first is the execution time, and the second is the possibility that the transaction commit fails as the processing process is complicated. If each vertex is regarded as an atomic operation, how to ensure that the final data is complete and accurate is the problem. That is, after a service data is triggered, some statistical data cannot be accurate, there is a small part of inaccuracy.

One solution to solve the data integrity of many atomic operations is to record the processing process of each atomic operation. If it is found that the processing is successful after the final processing is completed, it will be used Program To roll back the data that has been successfully modified. This solution has two disadvantages:
First, the rollback logic is complicated;
Second, if the computing service is unavailable, the data in the business system cannot trigger data computing, and the data cannot be involved in computing.

Solution: send data from the Business System to MSMQ. In the form of MSMQ, WCF will listen to MSMQ and process the message if it is received. Like the original solution, the results of each atomic data operation are recorded. If a failure is found, the message is sent to the exception message queue, then, specialized services are used to process the messages with computing failures. The Exception Processing Service can differentiate which messages need to be re-computed based on the message processing logs, because some data has been computed successfully, this can avoid repeated computation. Similarly, because MSMQ can be stored in a disk, even if the computing service is not online at the time, these messages will be sent again after the service is started normally to avoid data loss.

Structure of the asynchronous computing framework:

1: asynchronous computing proxy, which is responsible for receiving data submitted by the business system, extracting task configuration information, and finally sending messages to different levels of computing services through soft load.
2: A group of computing services, which are divided into high, medium, low, and so on based on their priorities. The number of services with high priorities is the largest. For example, in this way, the number of services with high priority is 3, medium 2, and low 1. If there are 100 requests with a high priority, each service only needs to handle less than 40 requirements. If the medium priority is used for processing, each service needs to process 50 requests, the worst low priority needs to be handled 100 times.
3: The exception computing service processes messages with errors in computing in the previous computing service. If the message fails again, the message is persisted for manual resolution.
4: Persistent storage. MongoDB is used here. We can achieve the maximum performance advantage when recording a large number of logs.

Data Flow Diagram of the asynchronous computing framework: