Performance comparison of several go serialization libraries

This is a creation in Article, where the information may have evolved or changed.

The serialized library is often used in the environment of network transmission, RPC, database access, and its performance directly affects the performance of the whole product.
This article lists several high-performance go language serialization libraries and tests their performance with a simple, loop-free reference data structure.
Test code: Gosercomp at GitHub

03/14/2016 Update. Increase the performance comparison of Thrift/avro/gencode.

Serializers of the test

Based on Golang's own Encoding/json and encoding/xml , the following are some of the more well-performing serialization libraries.

• Encoding/json
• Encoding/xml
• Github.com/youtube/vitess/go/bson
• Github.com/tinylib/msgp
• Github.com/golang/protobuf
• Github.com/gogo/protobuf
• Github.com/google/flatbuffers
• Apache/thrift
• Apache/avro

Exclusion of serializers

Based on Alecthomas existing tests, the following libraries are not tested for performance reasons.

• Encoding/gob
• Github.com/alecthomas/binary
• Github.com/davecgh/go-xdr/xdr
• Github.com/ugorji/go/codec
• Labix.org/v2/mgo/bson
• Github.com/dedis/protobuf
• Gopkg.in/vmihailenco/msgpack.v2

Test environment

Add into the ./bin PATH environment variable.

• For github.com/youtube/vitess/go/bson , you may need to install goimports and codegen :

1234	go get github.com/youtube/vitess/go/bsongo get golang.org/x/tools/cmd/goimportsGo get github.com/youtube/vitess/tree/master/go/cmd/bsongenbsongen-file data. go -o bson_data. Go -type colorgroup

• For MessagePack , you need to install the library and take advantage go generate of the build-related classes:

12	go get github.com/tinylib/msgpgo generate

• For ProtoBuf , you need to install the PROTOC compiler, as well as the PROTOC library and generate related classes:

123	go get github.com/golang/protobuf/protogo get github.com/golang/protobuf/protoc-gen-go Go Generate

• For gogo/protobuf , you need to install the library and generate related classes:

123	go get github.com/gogo/protobuf/gogoprotogo get Github.com/gogo/protobuf/protoc-gen-gofast Go Generate

• For flatbuffers , you need to install the thrift compiler as well as the Flatbuffers library:

12	go get github.com/google/flatbuffers/gogo generate

• For thrift , you need to install the Flatbuffers compiler as well as the Thrift Library:

12	go get git.apache.org/thrift.git/lib/go/thriftgo generate

• For Avro , you need to install the Goavro library:

12	go get Github.com/linkedin/goavrogo generate

• For gencode , you need to install the Gencode library and use the tools of the Gencode library to produce the data objects:

12	Go go -schema=gencode.schema- Package Gosercomp

In fact, by go generate generating related classes, you can also generate from the command line, refer to data.go the comments in. But you need to install the relevant tools, such as thrift, and add them to the environment variable path

Run the following command test:

1	go test-bench=. -benchmem

Test data

All tests are based on the following struct, and the automatically generated struct, such as PROTOBUF, is basically consistent with this structure.

12345

type struct { id int' JSON: ' ID ' xml: ' id,attr ' "' Name string' JSON:" Name "xml:" Name "' Colors []string' JSON:" Colors "xml:" Colors "'}

Performance Test Results

The test results for the following performance are in my t430u/windows 10 test results.

1234567891011121314151617181920212223242526272829303132

benchmark _name iter Time/iter alloc bytes/iter allocs/iter--------------------------------------------------------------------------------------------------------------- ----------BenchmarkMarshalByJson-4 1000000 1795 ns/op 376 b/op 4 Allocs /opbenchmarkunmarshalbyjson-4 500000 3927 ns/op 296 b/op 9 Allocs/opbenc HmarkMarshalByXml-4 200000 8216 ns/op 4801 b/op marshalByXml-4 50000 26284 ns/op 2807 b/op Bson-4 500000 3258 ns/op 1248 b/op allocs/opbenchmarkunmarshalbybson- 4 1000000 1433 ns/op 272 b/op 7 allocs/opbenchmarkmarshalbymsgp-4 5000000 259 ns/op b/op 1 allocs/opbenchmarkunmarshalbymsgp-4 3000000 466 ns/Op b/op 5 allocs/opbenchmarkmarshalbyprotobuf-4 2000000 955 ns/op 328 b/op 5 allocs/opbenchmarkunmarshalbyprotobuf-4 1000000 1571 Ns/op 400 B/op allocs/opbenchmarkmarshalbygogoprotobuf-4 10000000 224 ns/op 1 allocs/opbenchmarkunmarshalbygogoprotobuf-4 2000000 828 ns/op 144 B/op 8 allocs/opbenchmarkmarshalbyflatbuffers-4 3000000 626 ns/op b/op 1 All Ocs/opbenchmarkunmarshalbyflatbuffers-4 100000000 10.4 ns/op 0 b/op 0 allocs/ Opbenchmarkunmarshalbyflatbuffers_withfields-4 3000000 493 ns/op b/op 5 allocs/opBenchmarkMarshalByThrift-4 2000000 840 ns/op b/op 1 Allocs/opbenchma RkUnmarshalByThrift-4 1000000 1575 ns/op b/op 6 Allocs/opbenchmarkmarsh AlByAvro-4 1000000 1330 ns/op 133 b/op 7 Allocs/opbenchmarkunmarshalbya Vro-4 200000 7036 ns/op 1680 b/op 20000000 66.2 ns/op 0 b/op 0 allocs/opbenchmarkunmarshalbygencode-4 5000000 258 ns/op b/op 5 allocs/op

The results of many tests are similar. From the results, messagepack,gogo/protobuf, and flatbuffers , these three excellent libraries are different in both serialization and deserialization, and are cross-lingual. In terms of convenience, you can choose messagepack and Gogo/protobuf , both of which are used by manufacturers. Flatbuffers is a bit anti-human, because it operates very low, and from the results, the performance of the serialization is almost. But it has the advantage that if you only need a specific field, you don't have to deserialize all the fields. From the result, the field is not deserialized with only 9.54 nanoseconds per call, because the field is converted from a byte array to the appropriate type only when it is accessed. So in a special scenario, it can improve the performance of N. But the faces of the serialized code are too ugly.

New additions to the gencode test, which performed quite well, and the resulting bytes were very small.

The official protobuf is serialized deserialized by reflection (reflect), so the efficiency is not very high. Gogo/protobuf, in the case of code generation, provides a marshal/unmarshal approach, which is not reflected, so it's much better than the official library.