Explore the consistency hash of C #

Read the catalogue:

1. Usage Scenarios
2. Algorithm principle
3. Virtual node
4. code example

Usage Scenarios

For Redis, for example, when the system needs to cache more content than the single-machine memory size, such as caching 100G of data, only 16G is available for stand-alone memory. In this time, we need to consider the partition of the cache data, that is, the 100G of data into a number of pieces smaller than the single-machine memory data. For example, in 10G, split 10 copies, stored on more than one machine node. But how is the data a more reasonable method? 、

F (Key)%n

Here the n=10 is configured, and the different keys are mapped to the corresponding machine based on the value remainder. A very simple way to solve the problem of multiple node key division method. However, the growth and reduction of data size is difficult to predict, if you need to add a cache server. Configuring N=11, you will find that the mapping relationships established by the remainder are confusing.  After mapping the disorder, a large number of keys fail to hit the correct node and need to be fully re-mapped. If you add nodes later, you will also encounter this problem.

    Servers = ['redis:6379'redis:6380'Redis : 6381']    =  servers[f (key)% servers.length]

Consistent hash (consistent hashing)

To reduce the impact of adding or removing server nodes, resulting in a large number of key misses. A more reasonable division method is proposed, which is also a consistent hash algorithm. Why is it more reasonable to look below?

Algorithm principle Space Attribution

We assume in our brains that each node calculates a value in the form of Chash (IP), and n machines have n values. The numerical space of a virtual ring is formed by connecting the values to the end.
For example there are 3 machines:

Servers =['redis:6379'redis:6380'Redis : 6381']. Chash (server[0]) ==  chash (server[1]) = =   chash (server[2]) = =  

The calculated value of the machine, in the virtual ring in a clockwise direction to determine the spatial attribution, get:

100~200 space belongs to 6379 tubes.
200~300 space belongs to 6380 tubes.
300+,100-space belongs to 6381 tubes.

Key coordinates

At this point there are 3 keys to be stored in Redis, respectively, Key1-key3. The numerical coordinates of 3 keys are calculated by the Chash function:

Chash (key1) =102chash (key2)=chash (key3)=

Spatial mapping

After finding the numerical coordinates of key, we know the mapping relationship between key and machine node. That is, key1 should be stored in the 6379,key3 store to 6381.

Adding nodes

Due to the increase in cached data, a new node 6382 needs to be added. Calculate the spatial value:

Chash (6382) = =  

So his position in the virtual ring is as follows:

It is learned that 6379, 6381 of the numerical space area without any change, their stored key can still hit the normal.
One of the advantages: the hit impact on an existing cache is small.
But the original 6380 area 200~300 was invaded by 6382. 6382 of the space value of 250 is precisely divided into half, that is, the area of the 200~250 6380 tubes, but the 250~300 of the region is the new tube. (Use simple numbers for example to differentiate, actually not so precise)
Two advantages: achieving fragmentation of data
Also brings the disadvantage is: originally stored in 6380 (250~300 this part) of the old cache data can not hit, to go to the new 6382 take. Therefore, the consistency hash does not completely solve this effect, can only be minimized.

removing nodes

Same as adding nodes. For example, take off the new 6382,250~300 area also control the original 6380 tube, of course 6382 this part of the cache is lost.

Virtual node

Although the consistency hash realizes the data sharding, but because the node is few, the key may be concentrated to a large number of the above, cause the cache distribution is uneven. Especially when there are only a few or more than 10 machine nodes.
In order to reduce this effect, the consistency hash algorithm proposes a solution of virtual node. That is, a physical machine node corresponds to multiple virtual nodes. Here a physical node is configured to correspond to 2 virtual nodes, and this should be:

6379={6379a,6379b}6380={6380a,6380b}6381={6381a,6381b}

This becomes 6 nodes (which can be configured more), and they are also arranged clockwise by value on the virtual ring. As the nodes become more numerous, the corresponding numerical regions become larger. The numerical space mapping of key is more discrete, and the uniform distribution of key is improved from probability.

It is necessary to calculate the value of the real node, and also to calculate the virtual node value, then the virtual ring numerical space is formed by the value of the dummy node. Each set of virtual node values, corresponding to a single physical node.

servers= ['redis:6379'redis:6380'Redis : 6381'];

The servers and virtual nodes are mapped to 6379={6379a,6379b} in the F function below, 6380={6380a,6380b},6381={6381a,6381b}

The values of the virtual nodes are calculated, and the numbers correspond to the physical nodes. That

= f (servers) ={['redis:6379', '],['redis:6379', ' 300 '] ....., ['redis:6381',[]]}; Chash (key1) = =102∈vservers[0] ...   Chash (Key3) = =350∈vservers[1]  

The virtual node makes the key distribution more balanced, but does not solve the impact of adding machines, deleting nodes.

code example

1: Use a dictionary to simulate virtual rings and add nodes.

2: Calculate the key value, which node should belong to the value space area.

3: Calculates the distribution frequency. The more virtual nodes you replicate, the more evenly the distribution is.

        Private Static ReadOnlysorteddictionary<ULONG,string> _circle =Newsorteddictionary<ULONG,string>(); Static voidMain (string[] args) {            intReplicas = -; AddNode ("127.0.0.1:6379", replicas); AddNode ("127.0.0.1:6380", replicas); AddNode ("127.0.0.1:6381", replicas); List<string> nodes =Newlist<string>();  for(inti =0; I < -; i++) {nodes. ADD (Gettargetnode (i+"Test"+ (Char) (i)); }            varCounts = nodes. GroupBy (n = n, n = =N.count ()).            ToList (); Counts. ForEach (Index= = Console.WriteLine (index. key+"-"+index.            Count ()));        Console.ReadLine (); }

Output:

127.0.0.1:6380-39
127.0.0.1:6381-29
127.0.0.1:6379-32

Value of the virtual ring:

The rest of the code:

  Public Static voidAddNode (stringNodeintrepeat) {             for(inti =0; I < repeat; i++)            {                stringidentifier = node. GetHashCode (). ToString () +"-"+i; ULONGHashcode =Md5hash (identifier); _circle.            ADD (hashcode, node); }        }         Public Static ULONGMd5hash (stringkey) {            using(varhash =System.Security.Cryptography.MD5.Create ()) {                byte[] data =Hash.computehash (Encoding.UTF8.GetBytes (key)); varA = Bitconverter.touint64 (data,0); varb = Bitconverter.touint64 (data,8); ULONGHashcode = a ^b; returnhashcode; }        }         Public Static stringGettargetnode (stringkey) {            ULONGhash =Md5hash (key); ULONGFirstnode =Modifiedbinarysearch (_circle.            Keys.toarray (), hash); return_circle[firstnode]; }        /// <summary>        ///calculates the value of the key to derive the spatial attribution. /// </summary>        /// <param name= "Sortedarray" ></param>        /// <param name= "val" ></param>        /// <returns></returns>         Public Static ULONGModifiedbinarysearch (ULONG[] Sortedarray,ULONGval) {            intMin =0; intmax = Sortedarray.length-1; if(Val < sortedarray[min] | | val >Sortedarray[max])returnsortedarray[0];  while(Max-min >1)            {                intMid = (max + min)/2; if(Sortedarray[mid] >=val) {Max=mid; }                Else{min=mid; }            }            returnSortedarray[max]; }

View Code

Explore the consistency hash of C #