Talk about serialization issues using Redis and memcached in. NET Core

Objective

In the use of distributed cache, it is inevitable to do such a step, the data is serialized and then stored in the cache.

The serialization of this operation, perhaps explicit or implicit, depends on whether the package being used helps us do such a thing.

This article uses Redis and memcached in the. NET core environment As an example, where Redis is mainly used Stackexchange.redis, memcached is mainly Enyimmemcachedcore.

Let's take a look at some of our commonly used serialization methods.

Common serialization methods

Perhaps it is common practice to serialize an object into a byte array and then use that array to interact with the cache server.

With regard to serialization, there are many algorithms in the industry, and the results of these algorithms in a sense are the two problems of speed and volume .

In fact, when the operation of distributed cache, we are actually more important to these two issues!

Under the same conditions, the speed of serialization and deserialization can determine whether the speed of execution can be faster.

The result of serialization, which is what we want to plug into memory, can save a lot of valuable memory if we can make it smaller.

Of course, the focus of this article is not to compare that sort of serialization method, but to introduce how to use it in conjunction with caching, and incidentally, some points that can be considered when using caching.

Here's a look at some common serialized libraries:

1. System.Runtime.Serialization.Formatters.Binary
2. Newtonsoft.json
3. Protobuf-net
4. Messagepack-csharp
5. ....

In these libraries

System.Runtime.Serialization.Formatters.Binary is in the. NET class library itself, so it's a good choice when you're not relying on third-party packages.

Newtonsoft.json should not have to say more.

Protobuf-net is a protocol buffers implemented by. Net.

Messagepack-csharp is an extremely fast messagepack serialization tool.

This kind of serialization of the library is also the author usually involved, there are some unfamiliar is not listed!

Before we begin, we define a product class, and the related operations are based on this class.

public class Product{    public int Id { get; set; }    public string Name { get; set; }}

Let's take a look at the use of Redis.

Redis

Before we introduce serialization, we need to know that in Stackexchange.redis, the data we want to store is in the form of Redisvalue. And Redisvalue is supported by string,byte[] and many other data types.

In other words, when we use Stackexchange.redis, the data that is stored in Redis needs to be serialized into the type supported by Redisvalue.

This is what was said earlier that requires an explicit serialization operation.

Take a look first. NET class library provided by the BinaryFormatter.

Serialized operations

using (var ms = new MemoryStream()){    formatter.Serialize(ms, product);                    db.StringSet("binaryformatter", ms.ToArray(), TimeSpan.FromMinutes(1));}

Deserialization operation

var value = db.StringGet("binaryformatter");using (var ms = new MemoryStream(value)){    var desValue = (Product)(new BinaryFormatter().Deserialize(ms));    Console.WriteLine($"{desValue.Id}-{desValue.Name}");}

It's quite simple to write, but this time running the code will prompt the following error!

That is, our product class has no tag serializable. The following is the product class plus [Serializable] .

Run again, it's already done.

And look at Newtonsoft.json .

Serialized operations

using (var ms = new MemoryStream()){    using (var sr = new StreamWriter(ms, Encoding.UTF8))    using (var jtr = new JsonTextWriter(sr))    {        jsonSerializer.Serialize(jtr, product);    }                    db.StringSet("json", ms.ToArray(), TimeSpan.FromMinutes(1));}

Deserialization operation

var bytes = db.StringGet("json");using (var ms = new MemoryStream(bytes))using (var sr = new StreamReader(ms, Encoding.UTF8))using (var jtr = new JsonTextReader(sr)){    var desValue = jsonSerializer.Deserialize<Product>(jtr);    Console.WriteLine($"{desValue.Id}-{desValue.Name}");}

Since Newtonsoft.json does not add serializable to the class we want to serialize, there is no mandatory requirement, so it can be removed or preserved.

It is quite smooth to run.

Of course, it can also be handled in the following way:

var objStr = JsonConvert.SerializeObject(product);db.StringSet("json", Encoding.UTF8.GetBytes(objStr), TimeSpan.FromMinutes(1));var resStr = Encoding.UTF8.GetString(db.StringGet("json"));var res = JsonConvert.DeserializeObject<Product>(resStr);

And look at protobuf .

Serialized operations

using (var ms = new MemoryStream()){    Serializer.Serialize(ms, product);    db.StringSet("protobuf", ms.ToArray(), TimeSpan.FromMinutes(1));}

Deserialization operation

var value = db.StringGet("protobuf");using (var ms = new MemoryStream(value)){    var desValue = Serializer.Deserialize<Product>(ms);     Console.WriteLine($"{desValue.Id}-{desValue.Name}");}

Usage seems to be the same.

But it's not so easy to run like this. The error prompts are as follows:

There are two processing methods, one is to add the corresponding attribute to the product class and attributes, and the other is to use Protobuf.meta to handle the problem at runtime. You can refer to the implementation of AUTOPROTOBUF.

The following is handled in the first way, directly plus [ProtoContract] and [ProtoMember] these two attribute.

Running again is the result of what we expect.

The final look at Messagepack, according to its description and comparison on GitHub, seems to be a lot more powerful than other serialized libraries.

It is also the default is to add the same as Protobuf MessagePackObject and Key the two attribute.

But it also provides a iformatterresolver parameter that allows us to choose.

The following is a method that does not need to add attribute to demonstrate.

Serialized operations

var serValue = MessagePackSerializer.Serialize(product, ContractlessStandardResolver.Instance);db.StringSet("messagepack", serValue, TimeSpan.FromMinutes(1));

Deserialization operation

var value = db.StringGet("messagepack");var desValue = MessagePackSerializer.Deserialize<Product>(value, ContractlessStandardResolver.Instance);

It is also normal to run at this point.

In fact, serializing this step is very simple for redis because it explicitly lets us handle it and then stores the results.

The 4 methods shown above, from the perspective of use, seem to be similar, there is no big difference.

If you compare Redis with memcached, you will find that the memcached operation may be slightly more complex than Redis's.

Let's look at the use of memcached.

Memcached

Enyimmemcachedcore Default has a defaulttranscoder
, for general data types (int,string, etc.) this article does not elaborate, but specifically describes the type of object.

In Defaulttranscoder, the serialization of data of type object is based on Bson.

There is another Binaryformattertranscoder that belongs to the default, and this is based on what we said earlier. NET class library comes with the System.Runtime.Serialization.Formatters.Binary.

Let's take a look at the two kinds of transcoder to use.

First, define the methods that initialize the memcached correlation and read and write the cache.

The initialization memcached is as follows:

private static void InitMemcached(string transcoder = ""){    IServiceCollection services = new ServiceCollection();    services.AddEnyimMemcached(options =>    {        options.AddServer("127.0.0.1", 11211);        options.Transcoder = transcoder;    });    services.AddLogging();    IServiceProvider serviceProvider = services.BuildServiceProvider();    _client = serviceProvider.GetService<IMemcachedClient>() as MemcachedClient;}

The transcoder here is that we want to choose that serialization method (for object type), if it is empty, use Bson, if binaryformattertranscoder is BinaryFormatter.

There are two notes to be aware of

1. After the 2.1.0 version, Transcoder is changed from the itranscoder type to the string type.
   
2. After the 2.1.0.5 version, it can be done in a dependency injection form without specifying a string type of transcoder.

The read-write cache operates as follows:

private static void MemcachedTrancode(Product product){    _client.Store(Enyim.Caching.Memcached.StoreMode.Set, "defalut", product, DateTime.Now.AddMinutes(1));    Console.WriteLine("serialize succeed!");    var desValue = _client.ExecuteGet<Product>("defalut").Value;    Console.WriteLine($"{desValue.Id}-{desValue.Name}");    Console.WriteLine("deserialize succeed!");}

Our code in the Main method is as follows:

static void Main(string[] args){    Product product = new Product    {      Id = 999,      Name = "Product999"    };        //Bson    string transcoder = "";    //BinaryFormatter    //string transcoder = "BinaryFormatterTranscoder";                InitMemcached(transcoder);    MemcachedTrancode(product);        Console.ReadKey();}

For the two kinds of transcoder, run up is still relatively smooth, in the use of Binaryformattertranscoder Remember to add to the product class [Serializable] is good!

Let's look at how to use Messagepack to implement memcached transcoder.

Inherit Defaulttranscoder here, then rewrite serializeobject,deserializeobject and deserialize these three methods.

public class MessagePackTranscoder : DefaultTranscoder{    protected override ArraySegment<byte> SerializeObject(object value)    {        return MessagePackSerializer.SerializeUnsafe(value, TypelessContractlessStandardResolver.Instance);    }    public override T Deserialize<T>(CacheItem item)    {        return (T)base.Deserialize(item);    }    protected override object DeserializeObject(ArraySegment<byte> value)    {        return MessagePackSerializer.Deserialize<object>(value, TypelessContractlessStandardResolver.Instance);    }}

Fortunately, Messagepack has a way for us to serialize an object directly into arraysegment , or we can put arraysegment Deserialize into a object!!

Compared to JSON and PROTOBUF, save a lot of operation!!

At this time, we have two ways to make use of this new defined Messagepacktranscoder.

Mode one: In use, we only need to replace the previously defined Transcoder variable (for the >=2.1.0 version).

string transcoder = "CachingSerializer.MessagePackTranscoder,CachingSerializer";

Note: If the use of a way to handle, remember to transcoder the spelling of the wrong, and to take the namespace, otherwise the creation of the transcoder will always be null, so go is Bson! The essence is activator.createinstance, should not explain more.

Mode two: processing by dependency Injection (for >=2.1.0.5 version)

private static void InitMemcached(string transcoder = ""){    IServiceCollection services = new ServiceCollection();    services.AddEnyimMemcached(options =>    {        options.AddServer("127.0.0.1", 11211);        //这里保持空字符串或不赋值，就会走下面的AddSingleton        //如果这里赋了正确的值，后面的AddSingleton就不会起作用了        options.Transcoder = transcoder;    });    //使用新定义的MessagePackTranscoder    services.AddSingleton<ITranscoder, MessagePackTranscoder>();        //others...}

Add a breakpoint before running, and make sure it's actually in the method we're overriding.

The final result:

Protobuf and JSON, here is not introduced, these two processing is more complex than messagepack. can refer to memcachedtranscoder this open source project, is also written by Messagepack author, although 5 years ago, but the same easy to use.

For Redis, when calling the set method, we explicitly serialize our values first, not so concise, so we will use them once in a package.

For memcached, although it is not necessary to explicitly serialize a call to the Set method, it is possible to implement a transcoder ourselves, which is a bit cumbersome.

We recommend a simple cache library to handle these issues.

Use easycaching to simplify operations

Easycaching is the author in my spare time writing a simple open source project, the main purpose is to simplify the operation of the cache, is also in continuous improvement.

Easycaching provides a selection of 4 serialization methods as described above:

1. BinaryFormatter
2. Messagepack
3. Json
4. Protobuf

If these 4 kinds are not satisfied with the demand, you can also write their own, as long as the implementation of Ieasycachingserializer This interface corresponding methods can be.

Redis

Before we describe how to use serialization, let's take a quick look at how it's used (demo with the ASP. NET Core Web API).

Add a Redis-related NuGet package

Install-Package EasyCaching.Redis

Modify Startup

public class Startup{    //...    public void ConfigureServices(IServiceCollection services)    {        //other services.        //Important step for Redis Caching               services.AddDefaultRedisCache(option=>        {                            option.Endpoints.Add(new ServerEndPoint("127.0.0.1", 6379));            option.Password = "";        });    }}

Then use it in the controller:

[Route("api/[controller]")]public class ValuesController : Controller{    private readonly IEasyCachingProvider _provider;    public ValuesController(IEasyCachingProvider provider)    {        this._provider = provider;    }    [HttpGet]    public string Get()    {        //Set        _provider.Set("demo", "123", TimeSpan.FromMinutes(1));        //Get without data retriever        var res = _provider.Get<string>("demo");        _provider.Set("product:1", new Product { Id = 1, Name = "name"}, TimeSpan.FromMinutes(1))        var product = _provider.Get<Product>("product:1");                  return  $"{res.Value}-{product.Value.Id}-{product.Value.Name}";      }}

1. When used, a dependency injection of the Ieasycachingprovider is performed on the constructor.
2. Redis defaults to BinaryFormatter for serialization.

How do we replace the new serialization method we want?

Use Messagepack as an example to install the package from NuGet first

Install-Package EasyCaching.Serialization.MessagePack

Then just add the following sentence to the Configureservices method.

public void ConfigureServices(IServiceCollection services){    //others..        services.AddDefaultMessagePackSerializer();}

Memcached

Let's also start with a simple look at how it's used (demo with the ASP. NET Core Web API).

Add a memcached NuGet package

Install-Package EasyCaching.Memcached

Modify Startup

public class Startup{    //...    public void ConfigureServices(IServiceCollection services)    {        services.AddMvc();        //Important step for Memcached Cache        services.AddDefaultMemcached(option=>        {                            option.AddServer("127.0.0.1",11211);                    });            }    public void Configure(IApplicationBuilder app, IHostingEnvironment env)    {        //Important step for Memcache Cache        app.UseDefaultMemcached();        }}

When used in a controller, it is identical to Redis.

It is important to note that in Easycaching, the default serialization method is not Bson in Defaulttranscoder, but rather binaryformatter

How do I replace the default serialization operation?

Also take Messagepack as an example, first install the package through NuGet

Install-Package EasyCaching.Serialization.MessagePack

The rest of the operation is the same as Redis!

public void ConfigureServices(IServiceCollection services){    //others..    services.AddDefaultMemcached(op=>    {                        op.AddServer("127.0.0.1",11211);    });    //specify the Transcoder use messagepack serializer.    services.AddDefaultMessagePackSerializer();}

Because in easycaching, there is a own transcoder, this transcoder to ieasycachingserializer injection, so only need to specify the corresponding serializer.

Summarize

First, take a look at the 4 types of serialized libraries mentioned in the article

System.Runtime.Serialization.Formatters.Binary in the use of the need to add [Serializable] , the efficiency is the slowest, the advantage is the class library inside, there is no need to refer to additional package.

Newtonsoft.json is more friendly to use, probably for many reasons, and does not require us to add some attribute to the already defined classes.

Protobuf-net may be a little cumbersome to use, can be defined in the class when the corresponding attribute can also be processed at runtime (to pay attention to processing subclasses), but its reputation is good.

Although the Messagepack-csharp can not add the attribute, but the time of no increase will also be a loss.

As for how to choose, it may depend on the situation!

Interested can run with benchmarkdotnet points, I also wrote a simple to reference: Serializerbenchmark

Second, in the cache operation, may be more inclined to "implicit" operation, you can directly throw an object in, you can directly take out an object, at least to facilitate the use of the party.

Third, when serializing operations, Redis is simpler than memcached.

Finally, if you are using easycaching, there are questions or suggestions you can contact me!

Example code for the first half of the section: Cachingserializer

Sample code for the second half: sample

Talk about serialization issues using Redis and memcached in. NET Core