Design of Multi-terminal data synchronization mechanism and terminal Data Synchronization Mechanism

Design of Multi-terminal data synchronization mechanism and terminal Data Synchronization Mechanism
Design of Multi-terminal data synchronization mechanism (I)

Intro

A multi-terminal data synchronization mechanism needs to be designed to meet the following requirements: 1. data Operations and synchronization on multiple terminals 2. during each synchronization, only the data to be synchronized is pulled, and the data cannot be lost. call as few server interfaces as possible

Synchronization Process

Overall synchronization process

I want to perform data synchronization in the way of Git data synchronization, so I put the Git synchronization process for design. First, each commit will have a version number, in addition, we should try to pull data from the server before each submission to ensure that the client data is up-to-date and then submit local modifications. Data conflicts may occur during data synchronization in Git mode. If there is a data conflict, the client must resolve the conflict.
In general, there are two major operations: one is to pull data from the server and the other is to push data updates to the server.
There is a data version table at the database level to store each commit. Each update adds a record to the version table after the update, updating a version, return the version number to the client. Each time an update is pulled from the server, the updated data is not only returned to the client, but also to the client, it is used for the next data synchronization on the client.

Finally, the server provides three interfaces.

After thinking twice, the following flowchart is finally produced:

 

Obtain the latest version number of user data from the server

Client callGetCurrentVersion()Interface, you need to pass a parameter that identifies the user account, in order to query the data information of a user. Query the latest data version based on the user account information and return it to the client. The client compares the latest data version with the local version based on the server version, if they are consistent, it indicates that the latest version is used to determine whether there are local modifications or modifications. Then, you can directly submit the changes, if the inconsistency is not the latest version, the server pulls the data to update the data and the version number before submitting local modifications (if any ).

Data pulling process from the server

Pulling updates from the server is troublesome. If several versions on a device are not updated, You need to merge the data of these versions. The specific problem and process will be mentioned later.

The basic process for pulling data from the server is as follows:

 

The client pulls data and updates the local data process.

Client callPullDataThe interface pulls local data to be modified from the server and each data entry corresponds to an operation status to update local data. The operation status corresponding to the data is returned from the server at the same time, the client processes the data according to the returned operation status. When returning the data, it also needs to return the latest data version number for updating the local data version of the client.

 

The client pushes updates to the server.

Client callPushDataThe API pushes updates to the server and submits the changes to the server. The server returns the operation status of each data to be modified on the client and determines whether the changes are successful.

Update data on the server

After the client pushes updates to the server, the server must process the updates. First, you need to determine whether the client version is the latest version. If it is not the latest version, the client is prompted to update the local data to the latest version before updating the data. After that, the client request data (A json string) needs to be deserialized into a list of request entities. If the conversion fails, the client request data is not processed if there is a problem, if the conversion is successful, process it down. Then, the request object list is traversed and different data operations are performed based on the Operation Type of the request data. After each data operation is completed, the corresponding operation status is set. After all the request data is updated, add a new version and set the version to the response.

 

The pitfalls I step on

Pull data version merge

When pulling data from the server, we need to consider the issue of merging the submitted data of multiple versions. Our data is simpler to directly update the original data, therefore, it does not involve Text Segmentation and merging. However, you also need to merge the modifications of several versions, such as adding new data, if two data entries are added for each version, four data entries are returned. If one version is added and the other version is deleted, the data is not returned to the client. In this case, we need to consider how to efficiently and accurately return the data to be updated on the client. here we need to mention the content of my version table. In addition to the version number, there are updaters in the version table, update time and each callPushDataThe Request Parameters and responses returned to the client's operation status set are converted to json strings stored in the database. After each data update, a new version data is inserted into the version table.

Solution 1:

The first method is to extract the successfully modified data from the version table, merge the modifications of multiple versions into a List, and then repeat them, if two identical data items need to be de-duplicated, You need to determine the specific de-duplicated items based on the Operation Type of each data item, which is roughly divided into four situations:

In this example, not only the operation type needs to be modified, but also the data content needs to be merged and the latest data needs to be returned.

Solution 2:

The second method filters data and determines the data operation type based on the relationship between version Update time and data creation time and update time, if the data is deleted, the data is modified and not deleted.

First, query the data whose Update Time is later than the version Update time corresponding to the local version. The data is all the data after the local version is updated, and then filter the data, there are four types of operations:

Filter and determine the operation type, and then return the data to the client.

Make a comprehensive comparison to determine the version merge plan

After analysis, the data operation in the first solution is very troublesome, and the data operation in the second solution is very small. The data operation can be judged and filtered at the database level, and the data accuracy is similar to that in the second solution, to solve the concurrency problem, you can lock the Push interface according to the user account when calling the Push interface. After a comprehensive analysis, the second solution is used.

Push Interface

When calling the Push interface, the local version number was not determined. If the client fails to call the interface in the set order, an unimaginable data disaster may occur, as an interface, there is no way to control the client call sequence. Therefore, the modified Push interface requires the client to pass a parameter of the Client Version number. If the data is not the latest version, the client is prompted to update the data to the latest version before submitting the data.

Inconsistent time

This problem is a self-built pitfall. When updating data, it takes the website server time, however, when a new version is added, the update time in the new parameter is the time of the database server. Because the database server and website server are not on one server, the time on the database server is several seconds slower than the time on the website server. As a result, some data is not pulled out when I pull data from the server, later, debug queried the data from the database and the version was correctly inserted. At last, the update time of each piece of data and the Update time of each version were recorded one by one, in this case, I found that the time was a bit incorrect. Then I checked my SQL statements and found that the update time of the newly added SQL version wasGETDATE()The SQL statement used to update data is a parameter that uses the Time of the website server .. I suddenly wanted to quit myself when I found the problem ...(

In the end

Finally, there must be still some shortcomings in this design. I hope that the great gods can give their own views and opinions, and I hope they can tell me where they are incorrect.