Several ways of data transfer between application systems

This paper is reproduced from Sina_blog (Siangzhang)

With the rise of SOA (service-oriented technology architecture) in recent years, more and more application systems are starting to design and deploy distributed.
The system turns from a single technology architecture into a service-oriented, multiple-system architecture. The business processes that can be completed between systems are implemented through multiple interactions between systems.
It's not about how to design an SOA architecture, it's about how the data is transferred between application systems.

Data transfer between application Systems has three elements:

transmission mode, transmission protocol, data format

The way of data transmission is generally the following:

1 socket mode, 2 ftp/file Sharing server mode, 3 database sharing data way, 4 message mode

1, Socket mode

The socket method is the simplest way to interact. is a typical C/s interaction mode.
One client, one server.
The server provides services through IP addresses and ports for service access.
The client uses the port specified by the connection server for message interaction.
Where the transport protocol can be a TCP/UDP protocol.
and the server and the agreed request message format and Response message format. As shown in Figure one:

At present, we commonly used HTTP calls, Java remote invocation, webserivces is used in this way, but the difference is the transmission protocol and message format.

The advantages of this approach are:

1 easy to program, Java currently provides a variety of frameworks, shielding the underlying communication details and data transfer details.

2 easy to control permissions. Through the Transport Layer protocol HTTPS, encrypt the transmitted data, make security improve

3 Universality is strong, whether the client is a. NET architecture, Java,python is OK. Especially the WebService specification, which makes the service more general

And the disadvantage of this approach is:

1 server and client must work at the same time, when the server side is not available, the entire data interaction is not carried out.

2 when the volume of transmission data is large, the network bandwidth is severely occupied, which may cause the connection timeout. The service becomes unreliable when the volume of data is interacting.

2, ftp/file Sharing server mode

For the interaction of large amounts of data,
This is the best way to interact with this file.
System A and System B Convention file server address, file naming rules, file content format, etc.
Data interaction by uploading files to a file server.

The most typical scenario is batch processing data: For example, System A makes the data to be processed to a file before 12 o'clock today, and system B processes the next day 1 o'clock in the morning, processing is completed, and the processing results are generated to a file, system a
The results were processed 12 o'clock.

This kind of situation often occurs in a is the thing processing system, to the response request is higher, is not suitable does the data analysis type the work, but system B is the backstage system, to the processing ability request is relatively high, is suitable does the batch task system.

The above is only to explain the data through the file interaction, the actual situation B completed the task, may be the way through the socket to notify a, not necessarily through the file mode.

Advantages of this approach:

1 in the case of large amount of data, can be transmitted through the file, will not time out, do not occupy network bandwidth.

The 2 scheme is simple, avoids the network transmission, the network protocol correlation concept.

Disadvantages of this approach:

1 is not very suitable for real time class business

2 must have a common file server, the file server there is a risk. Because the file may be tampered with, deleted, or there is a leak.

3 must agree the format of the file data, when changing the file format, the need for the various systems are synchronized to make changes.

3 Database sharing data methods

System A and System B exchange data by connecting the same table to the same database server. When system a requests system B to process the data, system a inserts a piece of data, System B
Select System A inserts data for processing.

The advantage of this approach is

1 compared to file mode transmission, because the same database used, the interaction is simpler.

2 because the database provides quite do operations, such as updates, rollback, etc. The interactive mode is flexible, and the data exchange of reliability can be made through the transaction mechanism of database.

Disadvantages of this approach:

1 when the system connecting B is more and more, because the database connection pool is limited, causes each system to allocate the connection to be not many, when the system more and more, may cause no available database connection

2 in general, systems from two different companies are less likely to open up their own databases to connect to each other because of the security implications

4 Message Mode

The Java Messaging Service is a typical way of implementing message data transfer.
System A and System B exchange data through a single messaging server. System A sends a message to the message server, and if system B subscribes to the message from system A, the message server pushes the message to B.
Mutually agreed message format can be.

There are many open source JMS messaging middleware on the market today, such as  ActiveMQ, OPENJMS, RABBITMQ.

The advantages of this approach

1 because JMS defines the specification, there are many open source message middleware that can be selected and more generic. It's relatively simple to access.

2 through the message mode is more flexible, can take synchronous, asynchronous, reliable message processing, message middleware can also be deployed independently.

Disadvantages of this approach

1 Learning about the basics of JMS, the specifics of messaging middleware, and the details of implementation are a bit of a learning cost for developers

2 in the case of large data volumes, messages can accumulate, resulting in message latency, message loss, and even message middleware crashes.

The following concrete to analyze a scene, to see the application of data transmission between the system

Scene
At present, the business staff need to import a large file to system A, system A to save the file information, and the details of the file needs to be imported into the system B analysis, when the system B analysis is completed, you need to inform the system A of analysis results.

A system a first saves the file to the file server.

B System A via WebService
Call the server provided by System B and send the file name that needs to be processed to system B. Because the file is very large, need to deal with a long time, so b not in time to deal with the file, but to save the file name to be processed to the database, through the background timing scheduling mechanism to deal with. So b receives the request successfully, returns immediately system a succeeds.

C system B-time query database records, through the record lookup file path, find the file for processing. This process is very long.

D system B processing completed after sending a message to system A, to inform the system a file processing completed.

E system A receives a message from System B requesting a display of the task results