Design of a data exchange processing method between heterogeneous systems

Description:

The data exchange between the M and E systems of two different business systems adopts the ESB platform, which ensures that the data transfer process inside the ESB can have a global transaction, if any exception occurs, it will be rolled back.

Data is transmitted from m to E.

As agreed in the earlier stage, the ESB obtains data from the M system. The data is obtained from the intermediate database table agreed by the ESB and M system (equivalent to the front-end host of the M system ), call the WebService of E system to write data to E system, and write the Call Success result back to the intermediate table agreed by the M system (the work of writing back the intermediate table of the M system is completed by the ESB). After the WS of the E system is called, in addition to the write-back and M system's intermediate tables, there are also some other database updates and other operations. (Monitoring and incremental data in the middle table of the M system, using database triggers, and recording incremental data information (for example, new time and other information) in the additional table. After all the operations are successful, delete the monitored incremental data from the middle table of the M system .)

Problem:

1. The problem of transactions can be handled by the ESB and ignored.
2. If the data of WS fails to be written to the E system, WS does not throw an exception, but returns a code that fails to be written and returns the cause.
3. WS is successfully called and written to E system. However, if an exception occurs when the ESB handles other tasks in the future, the transaction will still be rolled back. The ws of E system does not have a compensation mechanism and does not determine whether the data already exists. In addition, the WS of E system have poor performance during calling due to various reasons.

Solution:

For question 1. No need to pay attention.

For question 2. As the ESB and M system agreed in the early stage, once the WS of the E system called by the ESB fail to be written, the M system will generate new data instead of modifying the original data. The ESB only needs to write the WS results that call E system into the intermediate table. Regardless of any exception, the M system re-generates data and re-transmits the data. In this way, the ESB only needs to monitor new data and perform corresponding operations in sequence.

For question 3. Before calling the WS of E system, you must determine whether the corresponding data exists in E system. If yes, the returned results are normal in the last call, instead of repeatedly calling WS, you can directly perform subsequent operations. If the returned results do not exist or are not normal during the call, or even do not return the results, you can re-execute the corresponding operations from the beginning.

In addition, if there is no agreement with the M system regarding the failure to write data to the WS of the E system, the ESB should judge based on the WS return value of the E system. If it is abnormal, you can directly throw an exception to roll back the entire transaction (of course, You can temporarily Save the exception data to another database table or another location to prevent data duplication ), ensure that the process can obtain this value again next time. Of course, there is also a problem, that is, rolling back the entire transaction. In the intermediate table agreed with the M system, the information about whether the call results are normal cannot be obtained, unless there is a place on the ESB platform that allows you to view exception information and data, and there is a dedicated person to handle such a thing, it seems that the ESB does not move to a certain piece of data, will affect the business.

Another solution is that the incremental data of monitoring is recorded in a table. If the WS of system e fail to write data, then, the ESB is not allowed to delete the monitored incremental data. It does not throw an exception to roll back the entire transaction, so that the cause of the error can be viewed normally.

I personally think the final solution is a good idea. Record it to identify myself.