Google flatbuffers-a new-generation open-source and cross-platform serialization Tool

I just tried a serialization tool cereal some time ago. Please refer to the open-source serialization library implemented by cereal: C ++. I plan to summarize my use of the Google proto Buf serialization library,

As a result, Google has developed a new, open-source, cross-platform serialization tool:Flatbuffers. So simply first understand this tool.

1. What is Google flatbuffers?

Flatbuffers is an open-source, cross-platform, efficient serialization tool library that provides C ++/Java interfaces. It was created by Google specifically for game development or other performance-sensitive application needs. It is especially suitable for mobile platforms. The memory size and bandwidth on these platforms are limited compared with those on the desktop system, and applications such as games have higher performance requirements. It stores serialized data in the cache. The data can be stored in files and transmitted as is through the network without any parsing overhead.

Code hosting homepage: https://github.com/google/flatbuffers;

Project homepage: http://google.github.io/flatbuffers/index.html;

Ii. Why Google flatbuffers?

1. Access to serialized data does not require packaging or unpacking-It stores serialized data in the cache, which can be stored in files and transmitted as is through the network, without any parsing overhead;
2. Memory efficiency and speed-the only memory requirement for data access is the buffer zone. No additional memory allocation is required. The detailed benchmark test can be viewed here;
3. Scalability and flexibility-the optional fields it supports mean not only good forward/backward compatibility (especially for games with a long life cycle, because no new version is required to update all data );
4. Minimum Code dependency-only a small amount of code that needs to be automatically generated and a single header file dependency can be easily integrated into the existing system. Again, check the details of the benchmark;
5. Strong type design-try to make errors occur during the compilation period, instead of manually checking and correcting them during the runtime;
6. Simple to use -- the generated C ++ Code provides simple access and construction interfaces. If necessary, an optional function can be used to efficiently parse the schema and JSON-like text at runtime;
7. Cross-platform: supports C ++ 11 and Java without any dependent libraries. It works well on the latest GCC, clang, vs2010, and other compilers;

3. Why not use protocol buffers or JSON

Protocol buffers is indeed similar to flatbuffers, but the main difference is that flatbuffers does not need to parse/split packets before accessing data. Protocol buffers does not have the optional text import/export function, nor does it have schemas syntax features (such as union ).

JSON is very readable and convenient to use with Dynamic Language (such as JavaScript. However, when serializing data in a static type language, JSON not only has obvious disadvantages of low running efficiency, but also allows you to write more code to access data (this is contrary to intuition ).

For more "why" about flatbuffers, visit the flatbuffers White Paper.

4. How to Use

1. Write a schema file (also called IDL) that defines the data you want to serialize. The data type can be int, float, String, array, or reference of another object of various sizes, even an object set;
2. All data attributes are optional and default values can be set.
3. Use the flatbuffer compiler flatc to generate C ++ header files or Java classes. The generated code provides additional helper classes for accessing and constructing serialized data. The generated code only depends on flatbuffers. H. See how to generate the code;
4. Use the flatbufferbuilder class to construct a binary buffer. You can add various objects to the buffer cyclically, and it is very simple, it is a single function call;
5. Save or send the buffer
6. When you read the buffer again, you can get the pointer of the buffer root object, and then you can simply read the data content locally;

5. A simple Schemas (IDL) File

namespace zl.persons;    enum GENDER_TYPE : byte  {      MALE    = 0,      FEMALE  = 1,      OTHER   = 2  }    table personal_info  {      id : uint;      name : string;      age : byte;      gender : GENDER_TYPE;      phone_num : ulong;  }    table personal_info_list  {      info : [personal_info];  }    root_type personal_info_list;

Note: the differences between table and struct are as follows:
Table is the main way to define objects in flatbuffers. The biggest difference between table and struct is that each of its fields is optional (similar to optional field in protobuf ), all the members of struct are required.
In addition to the member name and type, table also provides a default value for the member. If it is not explicitly specified, the default value is 0 (or null ). Struct cannot define scalar members, such as members of the string type. When the C ++ code is generated, the struct member sequence is consistent with the definition sequence of IDL. If alignment is necessary, the generator automatically generates additional members for alignment. See the following schemas code:

struct STest{    a : int;    b : int;    c : byte;}

After being generated as C ++ code, two members _ padding0 and _ padding1 will be added for padding:

MANUALLY_ALIGNED_STRUCT(4) STest { private:  int32_t a_;  int32_t b_;  int8_t c_;  int8_t __padding0;  int16_t __padding1;  public:  STest(int32_t a, int32_t b, int8_t c)    : a_(flatbuffers::EndianScalar(a)), b_(flatbuffers::EndianScalar(b)), c_(flatbuffers::EndianScalar(c)), __padding0(0) {}   int32_t a() const { return flatbuffers::EndianScalar(a_); }  int32_t b() const { return flatbuffers::EndianScalar(b_); }  int8_t c() const { return flatbuffers::EndianScalar(c_); }};
STRUCT_END(STest, 12);

The table's member sequence is dynamically adjusted, which is different from struct. When C ++ code is generated, the generator automatically adjusts to the optimal order to ensure that it occupies the minimum memory space.

6. A complete demo

Only one function is provided to demonstrate how to serialize objects. For the complete project, click Download directly or go to GitHub to view google_flatbuffers_test.

std::string CreateOnePerson(){    flatbuffers::FlatBufferBuilder builder;    fb_offset<fb_string> name = builder.CreateString("hello word");    zl::persons::personal_infoBuilder pib(builder);    pib.add_id(1);    pib.add_age(25);    pib.add_gender(zl::persons::GENDER_TYPE_MALE);    pib.add_name(name);    pib.add_phone_num(1234567890);    flatbuffers::Offset<zl::persons::personal_info> personinfo = pib.Finish();    fb_offset<zl::persons::personal_info> info[1];    info[0] = personinfo;    fb_offset<fb_vector<fb_offset<zl::persons::personal_info>>> info_array = fb_create_vector(builder, info, sizeof(info) / sizeof(info[0]));    fb_offset<zl::persons::personal_info_list> info_list = create_personal_info_list(builder, info_array);    fb_finish(builder, info_list);    // return the buffer for the caller to use.    return std::string(reinterpret_cast<const char *>(builder.GetBufferPointer()), builder.GetSize());}

VII. Miscellaneous

In addition to benchmark tests published by Google, some people have tested and verified the performance. The above IDL file is derived from the author's article.

8. Reference

Http://google.github.io/flatbuffers/index.html

Http://powman.org/archives/md__schemas.html

Http://blog.csdn.net/menggucaoyuan/article/details/34409433

Http://liubin.org/2014/06/19/google-flatbuffers-cross-platform-serialization-library/