Introduction and preliminary use of FlatBuffers

Introduction and preliminary use of FlatBuffers

Overview ###

Google released the cross-platform serialization tool FlatBuffers in June this year, which provides support for C ++/Java/Go/C # interfaces. This is a serialization class library focusing on performance and resource usage. Compared with Protocol Buffers, FlatBuffers is more suitable for mobile devices and provides higher performance and lower resource requirements.

Features #### no package or unpackage is required. Its structured data is saved in binary format and does not require data parsing. data can also be easily transferred to systems with high memory performance, in the compilation phase, we can prevent the generation of some bugs. cross-platform (C ++ 11/Java/Go/C #) Comparison of FlatBuffers, Protocol Buffers, and Json: #### FlatBuffers features similar to Protocol Buffers. Their biggest difference is that before using specific data, FlatBuffers does not need to parse/unpackage. At the same time, the reference of FlatBuffers is much more convenient than that of Protocol Buffers in engineering. It only needs to contain two or three headers and JSON can be used as the data exchange format, it is widely used among various dynamic languages (including static languages, of course ). It is easy to understand (readable), and its biggest drawback is the performance problem during parsing. Because of its dynamic type characteristics, your code may need to write more types and Data check logic. How to Use FlatBuffers #### compile a schema (IDL, interface definition) file used to define the data structure, as shown below:

    //flatbuffers test struct    namespace Jason.Flat.Test;    enum Color : byte { Red = 1, Green, Blue }    union Any { TextureData, Texture }    table TestAppend {        test_num:int;        test_num2:int;    }    table TextureData {        image_size:int (id:0);        image_data:[ubyte] (id:1);        image_test:short(id:3);        test_num2:int(id:2);    }    table Texture {        num_textures:short;        textures:[TextureData];        num_test:short = 30;        num_test1:short (deprecated);            num_test2:short;        test_append:TestAppend;    }    root_type Texture;

Save the preceding code as the TestFlat. fbs file, and then useFlatc to compile

Use FlatBuffer compiler flatc to generate data structure Source Code (C ++ header file or Java class)
Log on to GitHub and download the source code and engineering files of the desired version. The build directory contains the VS2010 project files (you can also choose to use CMake to create your own local project ), open the input parameters of the configured flatc project, as follows:
-C-o././TestFlat. fbs
Run the flatc project to generate the TestFlat_generated.h header file under the current project directory. This header file contains all the struct and enumeration types we need, as well as the corresponding access methods and verification methods.

Use the FlatBufferBuilder class to create the flat binary buffer.
The following code creates a buffer using FlatBufferBuilder:

 //read serialized buffer flatbuffers::FlatBufferBuilder builder_data; int test_append = 300; auto name_test = builder_data.CreateString("TestAppend"); auto testApp = CreateTestAppend(builder_data, test_append, test_append); int image_size = 12; unsigned char inv_data[] = { 11, 2, 4, 2, 10, 3, 5 ,7, 10, 39, 45, 23 }; auto name = builder_data.CreateString("TextureData"); auto image_data = builder_data.CreateVector(inv_data, image_size); int image_test = 900; auto texture_data = CreateTextureData(builder_data, image_size, image_data, image_test, image_test); //flatbuffers::FlatBufferBuilder builder_tex; int texture_num = 1; auto name_tex = builder_data.CreateString("Texture"); vector  > tex_vec; tex_vec.push_back(texture_data); auto tex_data = builder_data.CreateVector(tex_vec); int num_text = 100, num_text2 = 200, num_text3 = 300; auto texture = CreateTexture(builder_data, texture_num, tex_data, num_text, num_text2, testApp); builder_data.Finish(texture); 

To make the above run correctly, in addition to referencing the C ++ basic library, you must add the following code in the file header:

#include "flatbuffers/flatbuffers.h"#include "idl.h"#include "util.h"#include "TestFlat_generated.h"using namespace Jason::Flat::Test;

The above code is well written. Both CreateString and CreateVector are member functions of the FlatBufferBuilder class, which are used to create string data and vector data suitable for the FlatBuffer memory structure. Other methods, such as CreateTextureData and CreateTexture, are functions used to create the corresponding struct in the header file automatically generated by flatc according to the IDL file (TestFlat. fbs. Last sentenceBuilder_data.Finish (texture) is used to optimize the memory structure for alignment writing to builder_data.

Save the buffer to the local device or directly send it over the network.
The code for saving the buffer to the local device is as follows:

std::cout << builder_data.GetSize() << std::endl;flatbuffers::SaveFile("texture.bin", reinterpret_cast  (builder_data.GetBufferPointer()), builder_data.GetSize(), true); 

Save the data to a binary file named texture. bin.Builder_data.GetBufferPointer () to get the memory pointer,Builder_data.GetSize () gets the memory size. The last parameter is used to determine whether to generate a binary file.

Receive and buffer and read data
The code for reading a binary file is as follows:

 string binaryfile; bool ok = flatbuffers::LoadFile("texture.bin", false, &binaryfile); flatbuffers::Verifier tex_verify(builderOut.GetBufferPointer(), builderOut.GetSize()); bool verify_flag = VerifyTextureBuffer(tex_verify); flatbuffers::FlatBufferBuilder builderOut; TextureBuilder* texBuilder = new TextureBuilder(builderOut); builderOut.PushBytes(reinterpret_cast  (const_cast  (binaryfile.c_str())), binaryfile.size()); std::cout << builderOut.GetSize() << std::endl; auto model = GetTexture(builderOut.GetBufferPointer()); int outNum = model->num_textures(); const flatbuffers::Vector  >* outTex = model->textures(); TextureData* outTexData = (TextureData *)outTex->Get(0); int outSize = outTexData->image_size(); const flatbuffers::Vector  * outData = outTexData->image_data(); int x = outData->Get(5); int len = outData->Length(); delete texBuilder;    

In the above Code, VerifyTextureBuffer is used to verify whether the READ memory is a FlatBuffers memory block. If yes, true is returned. If not, false is returned. After obtaining the pointer through GetTexture, all the variables in the struct can be obtained through the corresponding method (For details, refer to the automatically generated header file.

Summary ####

The advantage of using FlatBuffers for data storage and transmission is obvious. It uses its own special encoding format to reduce memory usage to a certain extent and optimize reading performance. More importantly, it provides good scalability for forward and backward compatibility of data structures, which is convenient and efficient:

To make the data structure scalable, You need to define the data structure as a table, which is the basis for data expansion, the struct type in FlatBuffers does not support expansion. If you want to delete some fields in the data structure in a later version, you only need to add (deprecated) after the field to be deleted, of course, it is necessary to ensure that the Deleted field will not cause program crash in the previous version of the program (the Deleted field will get a null pointer or null value in the previous version of the program, you only need to ensure that the program does not encounter any exception after obtaining the null value or NULL pointer. If you want to add some fields to the data structure in a later version, to add a field to the end of the last field in the table, if you want to add a field anywhere in the table, you need to specify the Add id for each field as defined in TextureData above: n (n starts from 0)