FlatBuffers vs Protocol Buffers, flatbuffersbuffers

FlatBuffers vs Protocol Buffers, flatbuffersbuffers

FlatBuffers was released last year. Recently, it was similar to the Protocol Buffers from Google. According to the official website, FlatBuffers (FB) mainly targets game development and performance-demanding applications. Compared with Protocol Buffers (PB), FB does not need to be parsed. Data access can be completed only through the serialized binary buffer.

FB has the following features:

1) Data access does not need to be parsed

Serialize the data into a binary buffer, and then access the data directly to read the buffer, so the reading efficiency is very high.

2) memory efficiency and high speed

-Data access is only in the serialized binary buffer, and no additional memory allocation is required.

-The data access speed is close to the native struct, and only one more unreference (based on the starting address and offset, and then the value)

3) Elasticity

Optional fields, supporting forward and backward compatibility

4) a small amount of generated code

Only one header file is needed

5) strong type

Error detected during compilation

6) ease of use

Parses schema and json text.

7) cross-platform

C ++/Java/Go requires no additional dependencies

FB implementation principle:

The serialized data through FB is a binary buffer, which contains various objects (structs, tables, vectors) and is organized through offset. Therefore, data can be accessed in-place (Starting address + offset), just like an array or struct.

In FB, a table represents a data structure. This table corresponds to a vtable to indicate the offset of each field in the buffer. Next, When you access a field, you can find the field based on the offset. Because FB directly accesses raw buffer, dirty data is obtained when data is damaged. FB ensures that untrusted buffer data is correct through Verifier.

The following tests pb and fb for performance comparison. The test scenario is very simple. It constructs N objects, serializes, writes files, and deserializes them. The structure of the test object is as follows:

message Sample {    required int32 id = 1;    required string name = 2;    required int32 ip = 3;    repeated int32 addr_list = 4;}

The test results are as follows:

Number of records	FlatBuffers	Protocol Buffers
10000	0.197/4.4 M/0.02	0.166/3.0 M/0.08
20000	0.383/8.8 M/0.03	0.248/6.4 M/0.168
30000	0.534/13 M/0.05	0.378/10 M/0.26
100000	1.937/44 M/0.144	1.314/38 M/0.8
200000	3.9/88 M/0.276	2.66/67 M/1.542
1000000	17.59/439 M/1.396	13.355/397 M/8.012

The three data types X/Y/Z correspond to the total serialization time, file size, and reverse sequence time, including the time for reading and writing files. We can see that the serialization time and size of PB are superior to that of FB, the serialization speed is about 30% faster, and the serialization data size is about 20% smaller. Because FB does not need to be parsed, deserialization is extremely fast.

Although the deserialization of FB is super fast, the data size is quite large (probably because it does not need to be parsed and directly accesses binary buffer, so data cannot be compressed ), the transmission or storage overhead may be offset by the deserialization time. It seems that the use of FB is still very limited, and it may be an Application Scenario on the Mobile End.

For simplicity, the test code calls write or read for each object, and the statistical time also includes the time. Compared with PB and FB, the overhead is basically the same. For specific code, see github.