Overview
Multi-terminal heterogeneity. Generally speaking, this is the most complicated situation of
data integration system. In the previous part, we will discuss the practices in multi-terminal homogeneous scenarios, and then gradually extend to the topic of the article, that is, multi-terminal heterogeneous scenarios.
Multi-end isomorphism
The so-called multi-terminal isomorphism means that there can be multiple terminal types, but the databases used are the same, such as MySQL. In this case, the MySQL data of the terminal needs to be synchronized to the unified data platform, and the database type of the unified data platform can be arbitrary.
Generally speaking, there are two ways to achieve:
Periodically synchronize the bin log of the terminal MySQL to the unified data platform, and the platform parses the bin log and writes it to the database. The parser is independent of the business code and vice versa;
The service code of the terminal and the unified data platform sends and receives data through a custom data packet protocol, and writes it into their respective databases after analysis;
In the first way, a separate parser is required, that is,
data synchronization and business logic are separated. Therefore, the business module needs to be notified when the data arrives. This is generally implemented through a message queue. When the parser writes the data into the database, At the same time, a message is sent to the message queue, and the business code responds in real time by subscribing to the message.
The second way is actually to put the parser logic into the business module, this time the message queue is omitted.
In comparison, the first method requires two more modules to be deployed, but the business module is more pure and has no data integration logic, but the business module still needs to receive messages and then process data. And if the platform also has data that needs to be synchronized to the terminal, and the platform is not the way to collect incremental data in bin log, the parser and receiver become more complicated. Therefore, from this perspective, the second method is more common and less troublesome. Is the first method useless? Actually not. If there are multiple independent business modules on the platform that need to synchronize data, in the second method, each module needs to be directly connected to the terminal and send and receive data. In this case, the two methods can be combined to obtain the third method. the way.
Heterogeneous
Multi-terminal heterogeneous, that is, each terminal is different, and the database used is different. For example, some terminals use MySQL, some terminals use Oracle, and others use SQL Server.
In this case, you only need to change the multi-terminal isomorphic solution.
At this time, each terminal system and platform adopts a unified data exchange protocol, and each terminal module and platform module has the function of encapsulating data and parsing the data protocol.
There is also a more complicated situation, that is, each terminal module is developed by different manufacturers, there is no unified data protocol, and the terminal service module will not directly transmit data through TCP/IP and platform modules. At this time, the platform still needs terminal data, what should I do? You can use third-party software to monitor the terminal and the database changes of the platform, capture the changed data and send it to the other party, and then synchronize it to the database after the other party analyzes it. In this case, it looks similar to the first solution we gave when multi-terminal isomorphism. The third-party synchronization software is an independent data transceiver and parser.
I have compared various data synchronization tools. Many so-called synchronization tools on the Internet are more converters, which convert the data of a certain database into a structure that can be recognized by another database at a time and import it into the target database. Incremental synchronization cannot be performed. In terms of supporting incremental synchronization, there is an open source project called SymmetricDS, which supports two-way incremental synchronization of heterogeneous data, and the capture of data changes is based on triggers. In the above scenario, we need to install SymmetricDS into each terminal system, as well as the platform side. Each database is regarded as a node (Node) in the SymmetricDS architecture, each node belongs to a group (Group), the data synchronization rules are based on the group, that is, the synchronization between the group and the group, the nodes in the group use The rules of the group.
After SymmetricDS connects to a database, it will insert dozens of tables into the database to record data changes, routing rules, group information, node information, and so on. These tables also need to be placed together with our business tables for triggers to work, so this method is intrusive to the business database. If you don’t like this approach, we can convert the database structure of each terminal to the corresponding structure of the platform-side database, and then create a new database on the platform-side, and synchronize the data of the terminal database to the database just created for the terminal. This newly created database is equivalent to the backup of the terminal database. In this case, the newly created backup database is separate from the business database on our platform. How does the business module know when data is synchronized from the terminal to the platform? Similarly, through the message queue. We need to install a plug-in for the SymmetricDS service installed on the platform. This plug-in is called when the terminal data is synchronized to the backup database. The function of the plug-in is to send the received data to the message queue. The business module obtains the data by subscribing to the message. Deal with it accordingly.
Each terminal has a corresponding backup database on the platform side. When the terminal has new data synchronized to the backup database, the SymmetricDS plug-in sends the message to the message queue, and then enters the business database after being parsed by the message processing service. Reversely, when the business module has data update, the corresponding data is also written to the message queue. At this time, the message processing service writes the data to the backup database. Because of the trigger, SymmetricDS is triggered to save the changed data to the data first. Table, and then synchronized to the terminal database, so that the two-way incremental synchronization integration of data is realized.
Backend integration
If your data is not multi-terminal, such as only back-end integration, you can consider message queues. I also suggest you to check out Alibaba Cloud’s data-enhanced data platform. Provides to synchronize RDS data to Max Compute and other storage systems, and then can be connected to DataV for data analysis and display. This is the one-stop big data service of DataPlus. Alibaba Cloud has a data synchronization tool called DataX, which does not seem to support multi-terminal incremental synchronization, and later evolved into a product called "data integration". I don't know how it is. If you know more or have other plans, you are also welcome to share with you in the comments!
Multi-terminal offline
Multi-terminal offline refers to multi-terminal offline use. This is because someone asked me a question in the previous paragraph: When swiping their card at the terminal, the response speed is relatively slow, because the network is connected and the experience is not good when the network is not good. Can it be improved by storing the data in the terminal at the same time, and then not connecting to the server when swiping the card, and allowing a certain amount of overdraft. Another problem is that this card can be swiped in multiple terminals. For example, after swiping in this window, you can immediately go to another window to swipe it.
We can first look at how to solve the problem when multi-terminal card swiping is not allowed. In this case, store a copy of the data in the terminal, and then store data such as the allowable overdraft amount and the overdraft amount in the database. When the user swipes the card, it can be used normally if the range is not exceeded. When the network condition is good, the data will be synchronized to the back-end server.
However, when multi-terminal swiping is allowed, the data may not be synchronized to the server after the first swipe, and the data will not be synchronized when swiping at another terminal for the second time. In this case, it depends on the specific hardware and software conditions. If possible, store the required information on the card, so that there is no problem in the second and third terminals. When each terminal synchronizes the card consumption information to the server side, the server side is designed to be similar to playing back a movie, and it is enough to replay the action of each card swiping together with the specific recorded data for processing. This is just an idea I gave based on the situation described by the other party. Maybe after learning more information, there will be a different answer.
