Golang Serialization Frame Duel-why Andyleap/gencode so fast?

This is a creation in Article, where the information may have evolved or changed.

I created a project Gosercomp for performance comparisons of the Go Language serialization/deserialization library on GitHub to compare the familiar serialization libraries of the go language ecosystem.
Performance is based on the Json/xml serialization library provided by the Go official library, which compares how much the third library can deliver.
Although some third-party libraries automatically generate Struct code, we use the following data structure as an example:

12345

type struct {Id int ' JSON: ' ID ' xml: ' id,attr ' msg: ' ID ' ' Name string ' json: ' Name ' xml: ' Name ' msg: ' name ' 'Colors []string' JSON: "Colors" xml: "Colors" msg: "Colors" '}

Which Colors is a slice. I did not test the case of struct nesting and circular referencing.

Currently, this project contains the following performance comparisons for several serialized libraries:

• Encoding/json
• Encoding/xml
• Github.com/youtube/vitess/go/bson
• Github.com/tinylib/msgp
• Github.com/golang/protobuf
• Github.com/gogo/protobuf
• Github.com/google/flatbuffers
• Apache/thrift
• Apache/avro
• Andyleap/gencode
• Ugorji/go/codec

For the implementation of a serialization library, performance is not so good if it is serialized and deserialized at run time by reflection, such as the JSON of the official library and the XML serialization method, so many of the high-performance serialization libraries provide serialization and deserialization methods at compile time by code generation. Below I will introduce the messagepack and gencode two kinds of high performance serialization libraries.

This project is inspired by the Alecthomas/go_serialization_benchmarks project.

For third-party serialization libraries, their data structures may be defined in their own form, such as Thrift :

1234567

namespace go gosercompstructthriftcolorgroup{ 1i32 0, 2string name, 3list<string> Colors,}

For example flatbuffers :

123456789

namespace Gosercomp; Table Flatbuffercolorgroup { cgid:0); name:1); colors:2); }root_typeflatbuffercolorgroup;

For protobuf :

12345678

Package Gosercomp; messageprotocolorgroup { requiredint321; Required string 2; repeated string 3;}

As you can see, the data structures of all the tests are consistent, containing three fields, one field of type int Id , one field of type String Name , and one for []string Type field Colors .

Test results

The complete test results can be seen here,

The following are the performance data for JSON, XML, Protobuf, Messagepack, Gencode:

12345678910111213141516

benchmark _name iter Time/iter alloc bytes/iter allocs/iter--------------------------------------------------------------------------------------------------------------- ----------BenchmarkMarshalByJson-4 1000000 1909 ns/op 376 b/op 4 Allocs /opbenchmarkunmarshalbyjson-4 500000 4044 ns/op 296 b/op 9 Allocs/opbenc HmarkMarshalByXml-4 200000 7893 ns/op 4801 b/op marshalByXml-4 100000 25615 ns/op 2807 b/op ProtoBuf-4 2000000 969 ns/op 328 b/op 5 Allocs/opbenchmarkunmarshalbyproto Buf-4 1000000 1617 Ns/op b/op 5000000 ns/op b/op 1 allocs/opbenchmarkunmarshalbymsgp-4 3000000 459 ns/Op b/op 5 allocs/opbenchmarkmarshalbygencode-4 20000000 66.4 ns/op 0 b/op 0 allocs/opbenchmarkunmarshalbygencode-4 5000000 271 ns/op 32 B/op 5 Allocs/op

It can be seen that JSON, XML serialization and deserialization performance is poor. To compare Messagepack has a 10x performance boost, and Gencode is much better than messagepack.

The realization of Messagepack

MessagePackis an efficient binary serialization format. It can exchange data formats directly in multiple languages. It can serialize objects in smaller formats, such as small integers, which can use less storage (one byte). For short strings, it only needs an extra byte to indicate.

is a 27-byte JSON data that can be represented with 18 bytes if Messagepack is used.

It can be seen that each type requires an additional 0 to n bytes to indicate (the size or length of the number dependent object). The above example 82 indicates that the object is a map with two elements (0x80 + 2), A7 representing a short-length string with a string length of 7. Represents C3 True, which represents C2 false, which C0 represents nil. 00The representation is a small integer.

The full format can be referred to the official specification.

Messagepack supports multiple development languages.

Off-topic, a newer RFC specification, cbor/rfc7049 (a concise binary object representation), defines a similar specification that can express more detailed content.

The recommended Go Messagepack Library is TINYLIB/MSGP, which has better performance than Ugorji/go.

TINYLIB/MSGP provides a code generation tool msgp that can generate serialized code for Golang structs, and of course your struct should define msg tags, as defined above in this article ColorGroup . The go generate code can be generated automatically, as generated in this project msgp_gen.go :

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051

PackageGosercomp//Note:this FILE was produced by the//MSGP CODE GENERATION TOOL (GITHUB.COM/TINYLIB/MSGP)// do not EDITImport("GITHUB.COM/TINYLIB/MSGP/MSGP")//Marshalmsg implements MSGP. Marshalerfunc(Z *colorgroup) Marshalmsg (b []byte) (O []byte, err error) {o = msgp. Require (b, z.msgsize ())//Map header, size 3//string "id"o =Append(O,0X83,0Xa2,0x69,0x64) o = msgp. Appendint (o, z.id)//string "name"o =Append(O,0XA4,0X6E,0X61,0X6D,0X65) o = msgp. AppendString (o, Z.name)//string "colors"o =Append(O,0Xa6,0X63,0x6f,0X6C,0x6f,0x72,0x73) o = msgp. Appendarrayheader (O,UInt32(Len(z.colors))) forXVK: =Rangez.colors {o = msgp. AppendString (o, Z.colors[xvk])}return}//Unmarshalmsg implements MSGP. Unmarshalerfunc(Z *colorgroup) Unmarshalmsg (BTS []byte) (O []byte, err Error) {varfield []byte_ = FieldvarIszUInt32Isz, BTS, err = MSGP. Readmapheaderbytes (BTS)ifErr! =Nil{return} forIsz >0{Isz--field, BTS, err = MSGP. READMAPKEYZC (BTS)ifErr! =Nil{return}SwitchMSGP. Unsafestring (field) { Case "id": Z.id, BTS, err = MSGP. Readintbytes (BTS)ifErr! =Nil{return} Case "Name":......

The generated code uses JSON and XML similar to the official library, providing methods for marshal and unmarshalmsg.

Combined with the Messagepack specification, you can see the MarshalMsg method is very concise, it uses the MSGP. The Appendxxx method writes the corresponding type of data to []byte, which you can pre-allocate/reuse []byte, which can be implemented zero alloc . You also notice that it also writes the name of the field to the serialized byte Slice, so the serialized data contains the object's metadata.

When deserializing, the name of the field is read, and the corresponding byte is deserialized to the corresponding field of the object.

Overall, the performance of Messagepack has been quite high, and the resulting data is very small, but also cross-language support, is worth the attention of a serialized library.

Gencode

Is there any room for improvement for Messagepack? Test data show that Andyleap/gencode performance is also better, and even performance is messagepack twice times.

Andyleap/gencode's goal is also to provide a fast and low-data serialization library.
It defines its own data format and provides tools to generate Golang code.

Here is the data format for my test.

12345

struct Gencodecolorgroup {Id int32name stringColors []string}

It provides a data type struct similar to Golang, a structure similar to the definition, and provides a set of data types.

You can generate the code for the data structure through its tools:

1	Gencode. EXE Go -schema=gencode.schema-package gosercomp

As with messagepack, for integers greater than or equal, 0x80 it is represented by 2 or more bytes.

But unlike Messagepack, it does not write the name of the field, that is, it does not contain the metadata of the object. At the same time, the extra data it writes contains only the length of the field and does not need to indicate the type of data.
All values are piloted with its length, and the object is compressed to save space like Messagepack, so its code is more straightforward and effective.

Of course, they are handled by the byte shift or copy directly copy the string, so the processing is also very efficient.

When deserializing, the values of each field are parsed in turn, because the type of each field is known at compile time, so the gencode does not need metadata and can intelligently manipulate bytes sequentially in the same way as the stream.

As you can see, gencode, in contrast to Messagepack, does not have the information to add additional metadata to the data, nor does it need to write the type information of the field, which can also reduce the resulting data size, and it does not deliberately compress small integers, short strings, small maps, Reduced code complexity and judgment branching, the code is more concise and efficient.

It is important to note that the code generated by Gencode does not rely on other third-party libraries in addition to the official library.

Judging from the test data, it has a better performance.