Application of consistent hashing algorithm in distributed cache

First, Introduction

About the consistency hash algorithm introduction There are many similar articles, need to turn some theories into their own knowledge, so with this article, this part of the implementation of this article also refers to some of the original methods.

This algorithm is commonly used in the host selection of distributed cache, see http://en.wikipedia.org/wiki/Consistent_hashing.

Second, the reason for the birth of the algorithm

Many large systems now rely on caching (k/v) (due to high concurrency and other factors such as database pressure (or disk IO) overload, need to cache to relieve pressure), in order to achieve good level of scalability,

Cache hosts do not communicate with each other (such as mencached), through the client computing key to get the data stored in the host node, the simplest way is to take the model, if:

----------------------------------------------------------------

Now there are 3 cache hosts, and now there is a key for cks data that needs to be stored:

Key = "Cks"

Hash (key) = 10

3 = 1---> represents the choice of the first host to store this key and the corresponding value.

Defects:

If a host is down or a host is added (it must be considered), the algorithm will result in a large number of cache failures (calculated to other hosts that do not cache the data), the database, etc. suddenly bear a huge load, it is likely to cause the DB service is not available and so on.

----------------------------------------------------------------

Three, the principle of consistent hashing algorithm

The algorithm needs to solve the problem of large amount of cache jitter when the host or outage is increased, the algorithm must have the following characteristics in the production environment:

1. Balance: The cache data is distributed to all cache hosts as evenly as possible, effectively utilizing the space of each host.

2. monotonicity: When there is an outage or a new host joins, try to protect the original cache can continue to hit, that is, a small number of caches need to be reassigned.

3. Load balancing: Each cache host is as balanced as possible, i.e. the distribution of key must be balanced in these hosts.

If we hash the key into int type (32 bytes), the value range is -2^31 to (2^31-1), we connect the values to form a circle, such as:

Suppose there are now 3 cache hosts: C01, C02, C03, put them on the ring (via IP hash, which is described later), such as:

If there are now 5 keys that need to be cached, they are a,b,c,d,e, assuming that they are distributed after hash, they are found in the nearest host, and stored on it:

If the new join node C04, the data distribution is as follows:

It can be found that only a small amount of cache is reassigned to the new host, reducing the stress caused by jitter.

But at the same time there is a problem, the data distribution is not as uniform as possible (when there are a large number of caches can be seen), it is necessary to the real cache node virtual multiple nodes, distributed on the ring,

When a virtual node is found in a clockwise direction and then mapped to a real node, you can know which host the data is cached on.

Iv. implementation of the algorithm (Java version)

The implementation of the algorithm has many, the following example is for reference only, the actual need to consider a number of other issues:

Suppose there are 4 hosts:

192.168.70.1-5

 Public classNode {PrivateString IP;
Represents the k/v stored in the hostPrivateConcurrentmap<object, object> map =NewConcurrenthashmap<object, object>();  PublicNode (String IP) { This. IP =IP; }     PublicString GetIP () {returnIP; }         PublicConcurrentmap<object, object>Getmap () {returnmap; } @Override PublicString toString () {returnIP; }}

Here is a scenario where no virtual nodes are implemented:

 Public classConsistenthash {//-2^31-(2^31-1) ring for storing nodes    Private FinalSortedmap<integer, node> circle =NewTreemap<integer, node>(); PrivateIhash Hashif;  PublicConsistenthash (Ihash hash) { This. Hashif =Hash; }         Public voidAddNode (node node) {circle.put (Hashif.hash (Node.getip ()), node); }         Public voidRemoveNode (node node) {Circle.remove (Hashif.hash (Node.getip ())); }         PublicNode getnode (Object key) {intHashcode =Hashif.hash (key); if(!Circle.containskey (hashcode)) {            //similar to getting the nearest storage node clockwiseSortedmap<integer, node> tailmap =Circle.tailmap (hashcode); Hashcode= Tailmap.isempty ()?Circle.firstkey (): Tailmap.firstkey (); }        returnCircle.get (hashcode); }}

Where Ihash is the hash method interface, the different hashing methods can be implemented, and the following is an int value based on the MD5 algorithm (there are other algorithms):

 Public Interface Ihash {        int  hash (Object key);    }

 Public classMd5hashimplImplementsIhash {MessageDigest Digest;  PublicMd5hashimpl ()throwsNoSuchAlgorithmException {Digest= Messagedigest.getinstance ("MD5"); } @Override Public intHash (Object key) {if(Key = =NULL)return0; inth =Key.hashcode (); byte[] bytes =New byte[4];  for(inti=3; i>-1; i--) {Bytes[i]= (byte) (H>> (i*8) ); }                 byte[] hashbytes; synchronized(Digest) {hashbytes=digest.digest (bytes); }                intresult = 0;  for(inti=0; i<4; i++) {            intIDX = i*4; Result+ = (Hashbytes[idx + 3]&0xff << 24)                    | (Hashbytes[idx + 2]&0xff << 16)                    | (Hashbytes[idx + 1]&0xff << 8)                    | (Hashbytes[idx + 0]&0xff); }        returnresult; }}

The test method is as follows:

 Public classConsistenthashtest { Public Static voidMain (string[] args)throwsException {consistenthash chash=NewConsistenthash (NewMd5hashimpl ()); //Nodeslist<node> nodes =NewArraylist<node>();  for(intI=1; i<5; i++) {node node=NewNode ("192.168.70." + i);//FakeNodes.Add (node);        Chash.addnode (node); } Map<string, set<integer>> counter =NewHashmap<string, set<integer>>();  for(Node n:nodes) {counter.put (N.getip (),NewHashset<integer>()); }                //distribution of random key testsset<integer> AllKeys = new hashset<integer> ();
Random random =NewRandom (); intTesttimes = 1000000;  for(inti=0; i<testtimes; i++) {            intRandomint =Random.nextint (); Node Node=Chash.getnode (Randomint); Set<Integer> count =Counter.get (Node.getip ()); Count.add (Randomint);
Allkeys.add (Randomint); }                 for(Map.entry<string, set<integer>>Entry:counter.entrySet ()) {System.out.println (Entry.getkey ()+ "\ T" +entry.getvalue (). Size ()+ "\ T" + (Entry.getvalue (). Size () *100/(float) Allkeys.size ()) + "%"); }    }}

Test results (actual results are different for each run):

IP Count Percent

--------------------------------------
192.168.70.1216845 21.6845%
192.168.70.47207 0.7207%
192.168.70.2749929 74.9929%
192.168.70.325891 2.5891%

-------------------------------------

The result is that each node is unevenly distributed, and each node needs to be virtualized as multiple nodes, and the Consistenthash algorithm changes as follows:

 Public classConsistenthash {//-2^31-(2^31-1) ring for storing nodes    Private FinalSortedmap<integer, node> circle =NewTreemap<integer, node>(); PrivateIhash Hashif; Private intVirtualnum;//virtual node as multiple nodes         PublicConsistenthash (Ihash Hash,intvirtualnum) {         This. Hashif =Hash;  This. Virtualnum =Virtualnum; }         Public voidAddNode (node node) { for(inti=0; i<virtualnum; i++) {circle.put (Hashif.hash (i+Node.getip ()), node); }    }         Public voidRemoveNode (node node) { for(inti=0; i<virtualnum; i++) {Circle.remove (Hashif.hash (i+Node.getip ())); }    }         PublicNode getnode (Object key) {intHashcode =Hashif.hash (key); if(!Circle.containskey (hashcode)) {            //similar to getting the nearest storage node clockwiseSortedmap<integer, node> tailmap =Circle.tailmap (hashcode); Hashcode= Tailmap.isempty ()?Circle.firstkey (): Tailmap.firstkey (); }        returnCircle.get (hashcode); }}

You only need to modify it inside the test function:

Here each node virtual 120, according to the actual situation to consider modifying the reasonable value, the virtual number is less than the partial uneven, a large number of trees will lead to the search efficiency is reduced, the two need to weigh.

Consistenthash chash = new Consistenthash (new Md5hashimpl (), 120);

---------------------------The distribution after the virtual node is added----------------------------------------

IP Count Percent
192.168.70.127791627.7916%
192.168.70.425143725.1437%
192.168.70.222664522.6645%
192.168.70.324387124.3871%

------------------------------------------------------------------------------------------------

Below, we simulate the cache failure rate below the outage and the new host situation:

 Public classHitfailuretest { Public Static voidMain (string[] args)throwsException {consistenthash chash=NewConsistenthash (NewMd5hashimpl (), 120); List<Node> nodes =NewArraylist<node>();  for(intI=1; i<5; i++) {node node=NewNode ("192.168.70." + i);//FakeNodes.Add (node);        Chash.addnode (node); } Set<Integer> AllKeys =NewHashset<integer>(); Random Random=NewRandom (); intTesttimes = 1000000;  for(inti=0; i<testtimes; i++) {            intRandomint =Random.nextint (); Chash.getnode (Randomint). Getmap (). Put (Randomint,0);        Allkeys.add (Randomint); }                //Remove host serial number        intREMOVEIDX = 1;                Chash.removenode (Nodes.get (REMOVEIDX)); intFailurecount = 0;  for(Integer key:allkeys) {if(!Chash.getnode (key). Getmap (). ContainsKey (Key) {Failurecount++; }} System.out.println ("Failurecount \ Percent"); System.out.println (Failurecount+ "\ T" + (Failurecount*100/(float) Allkeys.size ()) + "%"); }}

The results are as follows:

Failurecount Percent
23,166,923,.16,975%

The results show that there is a relatively low cache failure efficiency, and the lower the failure rate when the host is more.

V. Summary

The consistent hashing algorithm can better guarantee the availability and expansibility of the distributed cache, and most of the cache clients are implemented in this way (more factors are considered than above, such as performance problems, etc.).

The above implementation in the case of downtime or new machines when a small portion of the cache is lost, but in some cases the cache is not allowed to be lost, you need to do a cache backup, there are two ways:

1. Modify the client to ensure that the data is cached to two different machines, and any downtime data can still be found.

2. The backup is implemented by the cache server, and the nodes are backed up by using the non-fixed primary node (re-electing the oldest machine as the primary node when the main node fails).

Application of consistent hashing algorithm in distributed cache