Use Pipeline (PipeLine) and Bulk (batch) operations for Redis for significant performance gains

　　

The previous time in the user portrait, encountered such a problem, record a product of the user purchase group, just this need can be used in Redis Set,key as Productid,value

is the specific CustomerID collection, follow-up, I can check the ProductID to see if the CustomerID buy this product, if purchased, you can have relevant related recommendations, of course, this is only the system

A small business condition, this time I can use the Sadd operation method, the code is as follows:

        Static voidMain (string[] args) {Connectionmultiplexer Redis= Connectionmultiplexer.connect ("192.168.23.151:6379"); vardb =Redis.            Getdatabase (); varProductID =string. Format ("productid_{0}",1);  for(inti =0; I <Ten; i++)            {                varCustomerID =i; Db.            Setadd (ProductID, CustomerID); }        }

One: The problem

But the above code is obviously a big problem, Redis itself is based on a TCP request/response protocol mode, if not, you can use Wireshark to monitor:

As can be seen, there are many times the data between 192.168.23.1 and 192.168.23.151, from the transmission content can also be seen in a prefix called productid_xxx,

If there are millions of LAN such round trip, that this delay is conceivable, certainly not achieve the high performance we expect.

Second: Solution "Batch"

Just based on our existing business, I can periodically integrate batch ProductID and CustomerID, and then insert it into a specific product set in batch form,

Next I can change the code above to resemble the following:

1         Static voidMain (string[] args)2         {3Connectionmultiplexer Redis = Connectionmultiplexer.connect ("192.168.23.151:6379");4 5             vardb =Redis. Getdatabase ();6 7             varProductID =string. Format ("productid_{0}",1);8 9             varList =Newlist<int>();Ten  One  A              for(inti =0; I <Ten; i++) -             { - list. ADD (i); the             } -  -             db. Setadd (ProductID, list. Select (I-= (redisvalue) i). ToArray ()); -}

From the transmission of the request,response can be seen, this time we commit the past, greatly less in the network transmission of embarrassment.

Three: Ask questions again

The image of the product dimension we can solve, but we also have a CustomerID dimension, that is to say I need to maintain a set of CustomerID as key, where value is

The various averages of the CustomerID, such as "total trades", "Total transaction Amount" ... Wait for such aggregation information, and then push it over to a batch of CustomerID, which means you need to schedule

Maintain a small SIP set set, in which case a set of the bulk operation will be uncertain ... The original code is as follows:

1         Static voidMain (string[] args)2         {3Connectionmultiplexer Redis = Connectionmultiplexer.connect ("192.168.23.151:6379");4 5             vardb =Redis. Getdatabase ();6 7 8             //batch over data: Customeridlist, Ordertotalprice, specific business logic omitted9             varOrdertotalprice = -;Ten  One             varCustomeridlist =Newlist<int>(); A  -              for(inti =0; I <Ten; i++) -             { the Customeridlist.add (i); -             } -  -             //foreach updates the Set collection for each Redis +             foreach(varIteminchcustomeridlist) -             { +                 varCustomerID =string. Format ("customerid_{0}", item); A  at db. Setadd (CustomerID, ordertotalprice); -             } -}

IV: Solution "PipeLine"

The above code is of course not feasible in production, but in response to this problem, Redis has already proposed the relevant solution, that is the pipeline mechanism, the principle is still the same, the command set is integrated through

A request requests to send the past, by the Redis internal fake out a client to do the batch operation, the code is as follows:

1         Static voidMain (string[] args)2         {3Connectionmultiplexer Redis = Connectionmultiplexer.connect ("192.168.23.151:6379");4 5             vardb =Redis. Getdatabase ();6 7 8             //batch over data: Customeridlist, Ordertotalprice, specific business logic omitted9             varOrdertotalprice = -;Ten  One             varCustomeridlist =Newlist<int>(); A  -              for(inti =0; I <Ten; i++) -             { the Customeridlist.add (i); -             } -  -             varBatch =db. Createbatch (); +  -             foreach(varIteminchcustomeridlist) +             { A                 varCustomerID =string. Format ("customerid_{0}", item); at  - Batch. Setaddasync (CustomerID, ordertotalprice); -             } -  - Batch. Execute (); -}

Then, we look at the following wireshark, we can see a lot of sadd such a small command, this means that there are many commands are in the past, greatly improved performance.

Finally can look at the Redis, the data also has, is not very cool ~ ~ ~

192.168.23.151:6379> Keys *1)"Customerid_0" 2)"Customerid_9" 3)"customerid_1" 4)"Customerid_3" 5)"Customerid_8" 6)"customerid_2" 7)"customerid_7" 8)"customerid_5" 9)"Customerid_6"Ten)"Customerid_4"

Well, first of all, I hope this article will help you.

Use Pipeline (PipeLine) and Bulk (batch) operations for Redis for significant performance gains