Data Exchange Protocol in Software System Development

ArticleDirectory

• Open
• Internal Services
• Persistent Storage
• Cross-language
• Data Exchange Protocol can be upgraded
• Clever Use of compatible protocols
• Summary

In many places, there is the concept of "data exchange". The "Data Exchange" mentioned in this Article refers to the transfer of data to another system in a computer network. This is very similar to telling another person one thing.

When one person wants to tell another person one thing, we can communicate by phone, email, text message, Im tool or in a face-to-face manner. This method is similar to system data exchange through TCP, UDP, pipeline, and so on. When two people communicate, we need a common language to understand the meaning of each other. Similarly, to exchange data between the two systems, we also need to define an agreement that both parties understand, we call it "Data Exchange Protocol ".

Data Exchange Protocol

The purpose of the Data Exchange protocol is to allow correct data interaction between two systems. Therefore, almost all development languages provide easy-to-use data exchange functions. For example, a system developed using the Java language uses a MySQL database to store data, that is, data exchange with MySQL through the Mysql Data Exchange Protocol; the RMI of Java can facilitate distributed data exchange across machines. Rmi is also a data exchange protocol.

Generally, when we exchange data between different systems and languages, we will choose a Common Exchange Protocol or define an easy-to-use exchange protocol. WebService has been very popular. In the Web 2.0 era, lightweight rest protocols have been sought after again. So what protocol should we choose in our system?

How to select a Data Exchange Protocol

The protocol we choose has a lot to do with our application scenarios. We need to consider whether our development is convenient, whether interfaces are easy to publish, whether we need to consider bandwidth occupation costs, serialization and deserialization performance, interface protocol scalability, and so on. Next we will look at several common exchange protocol implementations.

Protocol	Implementation	Cross-language	Performance	Transmission volume	RPC
XML	Extensive	Almost all	Low	Very large	N (achievable)
JSON	Extensive	Large	Average	Average	N (achievable)
PHP serialize	Phprpc	Large	Average	Average	Y
Hessian	Hessian	Large	Average	Small	Y
Thrift	Thrift	Large	High	Small	Y
Protobuf	Protobuf	Large	High	Small	N (achievable)
Ice	Ice	Large	High	Small	Y
Avro	Apache Avro	Small	High	Small	Y
Messagepack	Messagepack	Large	High	Small	Y

The table above lists some features of some common data exchange protocols. There is no good or bad here, just to explain that different data exchange protocols are different, so we need to select in our application scenarios.

Open

Open platforms like Weibo and SNS and systems that provide JavaScript interface calls to static html pages all belong to this type. This type features:

• The caller is not completely controllable and is directed to the public network. You may not know who the caller is, what language it is, and how it is calling your data interface;
• Interface traffic is usually very large, requiring high performance and throughput;
• Security issues need to be considered, and the data submitted externally may not be legal.

In this case, you need to consider the bandwidth consumption of data transmission, the ease of use of data exchange protocols, and the degree of multi-language support. In the past, JavaScript interface calls for HTML pages were generally in XML format. In recent years, they have almost all been converted to JSON format, because JSON transmission is smaller and easier to use than XML. For open platforms, there are many application scenarios, So multiple exchange protocol formats are required. XML and JSON are basically provided. To improve the performance of the platform and the client, you can also provide the protobuf binary Exchange Protocol and increase compression support to save the performance consumption of bandwidth transmission and resolution.

Internal Services

For a large system, internal service data exchange is everywhere. From the basic and common database data exchange, memcached cache data exchange, Message Queue data exchange to the RPC service framework used between systems, and so on, it can be counted as the data exchange of internal services. Internal Services are characterized by a high speed (basically no bandwidth cost) without the need to consider firewalls or open to the outside ).

Data exchange protocols for internal services have a large selection space. Generally, consider the following:

• Performance of data exchange protocols
• Cross-language support required
• Message Body Size of the Data Exchange Protocol

Persistent Storage

For persistent storage, each data exchange protocol can actually be implemented. Generally, you need to consider the following based on the Application Scenario:

• Manual reading?
• Storage space consumption
• Serialization and deserialization Performance
• Compressed?

Cross-language

Assume that the front-end page display layer of our website is developed in PHP and the intermediate business logic is developed in Java. This involves cross-language data exchange. As long as the system is not simply using a language, you must consider this issue. In fact, considering future expansion and demand changes, it is best to consider cross-language data interaction protocols.

Data Exchange Protocol can be upgraded

When selecting the Data Exchange Protocol, we also need to consider what is similar to the database table? Scalability during schema design. For example, a data exchange protocol interface that provides user information now contains the user name, gender, and address information. During the upgrade, an IP address information for the last logon is added. If you do not consider the impact of the Data Exchange Protocol upgrade, it is likely that the previous client encountered an exception or the old data could not be correctly parsed.

Clever Use of compatible protocols

The clever use of compatible protocols is very useful. New Product compatibility provides existing mature data exchange protocols, which can reduce the use threshold and product development speed. For example, Sina's open-source memcacheq uses the memcached protocol.

Summary

There are many common open-source implementations of the Data Exchange Protocol. The data exchange protocol is just a very broad saying. In fact, as long as data serialization and deserialization are realized, it can be said that it is a protocol that can exchange data. The performance of the Data Exchange Protocol is actually the serialization and deserialization performance. If RPC is added, the performance of the RPC implementation is also very significant.

My Sina Weibo @ sunli1223. Welcome to discuss it with me.

Reference link:

• Performance Comparison of Java serialization and deserialization: https://github.com/eishay/jvm-serializers/wiki
• Java cross-language call Implementation Scheme http://rdc.taobao.com/team/jm/archives/389

About the author

Sun Li is currently a senior System Architect at qunar.com. He once worked for Phoenix, ku6, and Sohu. Years of Internet experience andProgramDevelopment: rich experience in distributed search engine development, high concurrency, big data website system architecture optimization, high availability, scalability, distributed system cache, database sharding, etc, it also has experience in O & M monitoring and automated O & M control. Is the author of phplock and phpbuffer, an open-source project. Recently, I developed a nosql database storage inetdb, which is a nosql database enthusiast.

This article has been first published on the infoq Chinese site. All Rights Reserved. The original Article is 《
The data exchange agreement in software system development. If you need to reprint it, please attach this statement. Thank you.

Infoq Chinese site is an online independent high-end technical staffCommunity, For Java ,. net, Ruby, SOA, agility, architecture and other fields to provide timely and in-depth information, high-end technology conferences such as qcon, offline technology exchange activities qclub, free mini book download such as architect, etc..