Serialization comparison of thrift, protocol buffers and msgpack

What is serialization?

Serialization is the process of converting the object state to a format that can be maintained or transmitted. In contrast to serialization, deserialization converts a stream into an object. These two processes can be combined to easily store and transmit data.

Why serialization?

1. Make the custom object persistent in a certain storage form;

2. transfer an object from one place to another.
3. Make the program more maintainability
The above is taken from Baidu encyclopedia.

Generally, data needs to be synchronized during network service architecture. To ensure data availability, data must be serialized before data transmission. Data transmission between multiple services involves several issues that may affect architecture selection:

• Efficiency
• Interface coupling (that is, whether an interface is upgraded on one server, whether other servers need to perform synchronization upgrade, and whether the old server is compatible with the upgraded data interface)
• Language scalability
• Data structure support

Efficiency is an important factor in measuring network service efficiency. Therefore, you need to carefully consider this issue. For efficiency comparison, there is a lot of data on the Internet. Here, I still did a simple test.

Interface coupling
For interface coupling, only thrift and Protocol buffers are described here, because msgpack is determined to be coupled. If the interface definitions of the server and the client are inconsistent, a read/write error is inevitable. msgpack is not described here.

• Thrift ☆☆☆
  To test whether the added interface is still available, I wrote the following test code:
  

  struct Person{                                                                                                                                       1:i64 id,         2:string name}struct Student{        1:i64 id,         2:string name,        3:string school}struct Student2{        1:i64 id,         2:string school,        3:string name}struct Student3{        1:i64 id,         2:double fee = 10000.0,        3:string school,        4:string name}

  The above is the thrift definition file. Assuming that person is an old interface, student series are several interfaces for upgrade.
  The code for serialization and deserialization is as follows:

      using apache::thrift::transport::TMemoryBuffer;    using apache::thrift::protocol::TBinaryProtocol;    using boost::shared_ptr;    shared_ptr<TMemoryBuffer> strBuffer(new TMemoryBuffer());    shared_ptr<TBinaryProtocol> binaryProtcol(new TBinaryProtocol(strBuffer));    Student3 a;    a.id = 10;     a.name ="holla back a";    a.school = "pku";                                                                                                                            a.write(binaryProtcol.get());    std::string serialized = strBuffer->getBufferAsString();    shared_ptr<TMemoryBuffer> strBuffer2(new TMemoryBuffer());    shared_ptr<TBinaryProtocol> binaryProtcol2(new TBinaryProtocol(strBuffer2));    strBuffer2->resetBuffer((uint8_t*)serialized.data(), serialized.length());    Person a2;     a2.read(binaryProtcol2.get());

  The above is the core test code. The test method is to send different student data and check whether the received person data is normal.
  The test results are as follows:
  Student and person can be completely transmitted. After student2 is passed to person, the name field of person corresponds to the school field of student2; after student3 is transmitted to person, the name field is null (the corresponding position is double ).
  Based on the above test results, we can conclude that:
  If the data sequence and type of the original interface are not changed, adding data at the back end of the structure does not affect the communication between the two ends.
  As for the principle, you can view the code generated read and write method source code, this document also has some discussion http://www.cnblogs.com/egmkang/archive/2011/05/14/2046335.html
  P.s. Thrift supports nested data type structures. For example, a new data type can be defined as follows:

  struct Company{    1:i64 id,    2:string name,    3:list<i64>ids,    4:set<i64>iSet,    5:map<i64, string> iMap,    6:Student s//OK} 

• Protocol buffers ☆☆☆☆☆
  Because the Protocol Buf contains the required, optional, and repeated fields, protocol Buf is a serialization method with the lowest interface coupling.

Language scalability

• Thrift ☆☆☆☆☆
  

  Available generators (and options):  as3 (AS3):    bindable:          Add [bindable] metadata to all the struct classes.  c_glib (C, using GLib):  cocoa (Cocoa):    log_unexpected:  Log every time an unexpected field ID or type is encountered.  cpp (C++):    cob_style:       Generate "Continuation OBject"-style classes.    no_client_completion:                     Omit calls to completion__() in CobClient class.    templates:       Generate templatized reader/writer methods.    pure_enums:      Generate pure enums instead of wrapper classes.    dense:           Generate type specifications for the dense protocol.    include_prefix:  Use full include paths in generated files.  csharp (C#):  delphi (delphi):    ansistr_binary:  Use AnsiString as binary properties.  erl (Erlang):  go (Go):  hs (Haskell):  html (HTML):  java (Java):    beans:           Members will be private, and setter methods will return void.    private-members: Members will be private, but setter methods will return 'this' like usual.    nocamel:         Do not use CamelCase field accessors with beans.    hashcode:        Generate quality hashCode methods.    android_legacy:  Do not use java.io.IOException(throwable) (available for Android 2.3 and above).    java5:           Generate Java 1.5 compliant code (includes android_legacy flag).  javame (Java ME):  js (Javascript):    jquery:          Generate jQuery compatible code.    node:            Generate node.js compatible code.  ocaml (OCaml):  perl (Perl):  php (PHP):    inlined:         Generate PHP inlined files    server:          Generate PHP server stubs    autoload:        Generate PHP with autoload    oop:             Generate PHP with object oriented subclasses    rest:            Generate PHP REST processors    namespace:       Generate PHP namespaces as defined in PHP >= 5.3  py (Python):    new_style:       Generate new-style classes.    twisted:         Generate Twisted-friendly RPC services.    utf8strings:     Encode/decode strings using utf8 in the generated code.    slots:           Generate code using slots for instance members.    dynamic:         Generate dynamic code, less code generated but slower.    dynbase=CLS      Derive generated classes from class CLS instead of TBase.    dynexc=CLS       Derive generated exceptions from CLS instead of TExceptionBase.    dynimport='from foo.bar import CLS'                     Add an import line to generated code to find the dynbase class.  rb (Ruby):  st (Smalltalk):  xsd (XSD):

  Let's just move on to the thrift command. All the thrift commands are supported...

• Msgpack ☆☆☆☆☆☆
  Ruby, Python, Perl, C/C ++, PHP, C #, Java, JS... almost not supported, you can refer to the home page http://msgpack.org/
• Protocol buffer ☆☆☆
  Compared with the two scary guys, Java, C ++, and Python