MessagePack: A New deserialization scheme, messagepack serialization

MessagePack: A New deserialization scheme, messagepack serialization

When I was learning redis, I saw a brief introduction to MessagePack and found it very interesting. So I took some time to get to know about it.

MessagePack introduction:

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple extensions ages like JSON. But it's faster and smaller.
Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

MessagePack is an efficient binary serialization format. It allows you to exchange data between different languages like JSON. But it is faster and smaller than JSON. A small integer is encoded into one byte. A short string only needs a byte larger than its length.

Supported languages:

MessagePack is supported by over 50 programming versions and environments.

File comparison after MessagePack and JSON Compression

 

Speed comparison of MessagePack, protocol buffers, and json

This figure is the homepage image of the previous MessagePack official website. The comparison of numbers does reflect the problem. Here we only discuss JSON and MessagePack.

Why is it small? Json:

{“name“:”heyue“,”sex“:”\u7537“,”company“:”sina“,”age“:30} 

This json is 57 bytes in length, but has to be added to represent the data structure (all marked in red ), it uses 23 bytes (that is, braces, quotation marks, colons, and so on ).

Core compression methods of MessagePack:

1. true, false, and so on: these are too simple. directly give 1 byte, (0xc2 indicates true, 0xc3 indicates false)

2. do not represent the length: numbers, and so on. They are naturally fixed-length. They use a byte to indicate what is next, for example, using (0xcc to indicate the end, is a uint 8, oxcd indicates that the latter is a uint 16, and 0xca indicates that the latter is a float 32 ).

3. uncertain length: for example, strings and arrays, the type is appended with 1 ~ Four bytes are used to store the length of a string. If the length of a string is less than 256, only one byte is required. The longest string that MessagePack can store is (2 ^ 32-1) the maximum size of a 4G string.

4. ext structure: indicates the specific small single metadata.

5. Advanced structure: MAP structure, that is, the data in the key => val structure, which is similar to the array, plus 1 ~ The four bytes indicate the number of items that follow.

This is the official data representation structure document: https://gist.github.com/frsyuki/5432559

In general, MessagePack optimizes numbers, multi-byte characters, arrays, and so on, reducing useless characters and binary formats, it also makes sure that no extra storage space is added because the MessagePack is smaller than JSON, So it depends on your data. If you store English strings, there is almost no difference ....

Why is it faster?

Let's talk about how to parse JSON. We usually use the cJSON library in development. The cJSON database is stored in a linked list, and its access method is similar to a tree. Each node can have siblings and can be searched through the next/prev pointer. It is similar to a two-way linked list. Each node can also have a child node and access it through the child pointer to enter the next layer. The problem is, when constructing this linked list, match the character with one character. You have to judge whether it is a pair of quotation marks, Parentheses, and so on...

However, MessagePack is much simpler and goes through it over and over again. From the front data header, you can know what data is behind and how much the pointer should move backwards, A lot of comparison procedures are missing from the construction of a linked list in JSON format.

 

MessagePack is mainly used to cache and store structured data:

1. Memcache exists, because it is smaller than json, it can save some memory, and the speed is faster than json, and the page speed is naturally faster. Of course, there is also a situation where I store json in mc and then directly output the available json on the page without parsing json (of course this is rare in actual development ).

2. There is a Key-val storage that can be persisted.

 

. NET Usage

1. Download the source code compilation and generate a DLL from GIT https://github.com/msgpack/msgpack-cli.git

2. Example

using System.IO;using MsgPack.Serialization;namespace MsgPack{class Program{static void Main(string[] args){CreateMsgPack();}static void CreateMsgPack(){WriteToFile();ReadFromFile();using (var stream = new MemoryStream()){var serializer = MessagePackSerializer.Create<Person>();serializer.Pack(stream, CreateIris());stream.Position = 0;var person = serializer.Unpack(stream);}}static void WriteToFile(){var serializer = MessagePackSerializer.Create<Person>();using(Stream stream = File.Open(@"C:\Users\Iris\msg.txt", FileMode.Create)){serializer.Pack(stream, CreateIris());}}static void ReadFromFile(){var serializer = MessagePackSerializer.Create<Person>();using (Stream stream = File.Open(@"C:\Users\Iris\msg.txt", FileMode.Open)){var iris = serializer.Unpack(stream);}}static Person CreateIris(){return new Person{Age = 28,Name = "Iris Classon",FavoriteNumbers = new[] {2,3,4}};}}public class Person{public string Name { get; set; }public int Age { get; set; }public int[] FavoriteNumbers { get; set; }}}