The application of consistent hash algorithm in memcached

Objective

Everyone should know that memcached to achieve distributed only in the client to complete, the current is more popular through the consistent hash algorithm to achieve. The general approach is to balance the hash value of the server with the total number of servers in the server, i.e. hash%n, The disadvantage of this approach is that when you increase or decrease the server, there will be more caches that need to be reassigned and will cause the cache to be unevenly distributed (it is possible that one server is allocated a lot, but few others).

Today, we share a consistent hash algorithm called "Ketama", which effectively suppresses the uneven distribution of caches through the concept of virtual nodes and different cache allocation rules, and minimizes the redistribution of caches when servers increase or decrease.

Implementation ideas

Suppose we now have n memcached server, if we use uniform rules to memcached set,get operation. So that objects with different keys are evenly distributed across these servers, and the get operation takes the same rule to the corresponding server to remove object, so that the individual servers are not the same as a whole?

What kind of a rule is that?

As shown, we now have 5 (a,b,c,d,e) memcached servers, which we link together to form a ring, each server represents a point on the ring, each point has a unique hash value, and the ring has a total of 2^32 points.

So how do you determine which point each server is specifically distributed on? Here we use "Ketama" hash algorithm to calculate the hash value of each server, get the hash value can be corresponding to the ring point. (You can use the server's IP address as the key for the hash algorithm.)

The advantage of this is that when I add server F, I just need to reassign the hash value between C and F to F on the original d, and the cache on the other server does not need to be reassigned. And the new server can help cushion the pressure of other servers in time.

In this case, we have solved the drawback that large caches need to be redistributed when adding or removing servers. So how to solve the problem of uneven allocation of cache? Because our server now occupies only 6 points on the ring, and there are a total of 2^32 points on the ring, it is extremely easy to cause a large number of hotspots on one server, A situation where there are few hot spots on a stage.

The concept of "virtual node" is a good solution to this problem of unbalanced load. By dividing each physically-existing server into N virtual server nodes (n is usually based on the number of physical servers, there is a good threshold of 250). In this way, each physical server actually corresponds to n virtual nodes. There are more storage points, and the load of each server is naturally balanced. Like the subway station exits, the more exports, the less crowded each outlet will be.

Code implementation:

1 //Save all virtual node information, key: Virtual node hash key, Value: virtual node corresponding to the real server2         Privatedictionary<UINT,string> hostdictionary =Newdictionary<UINT,string>();3         //Save hash key for all virtual nodes, sorted in ascending order4         Private UINT[] Ketamahashkeys =New UINT[] { };5         //Save the real server host address6         Private string[] Realhostarr =New string[] { };7         //number of virtual nodes per real server8         Private intVirtualnodenum = -;9 Ten          PublicKetamavirtualnodeinit (string[] hostarr) One         { A              This. Realhostarr =Hostarr; -              This. Initvirtualnodes (); -         } the  -         /// <summary> -         ///initializing a virtual node -         /// </summary> +         Private voidinitvirtualnodes () -         { +Hostdictionary =Newdictionary<UINT,string>(); Alist<UINT> Hostkeys =Newlist<UINT>(); at             if(Realhostarr = =NULL|| Realhostarr.length = =0) -             { -                 Throw NewException ("cannot pass in an empty server collection"); -             } -  -              for(inti =0; i < realhostarr.length; i++) in             { -                  for(intj =0; J < Virtualnodenum; J + +) to                 { +                     byte[] namebytes = Encoding.UTF8.GetBytes (string. Format ("{0}-node{1}", Realhostarr[i], J)); -                     //call Ketama hash algorithm to get hash key the                     UINTHashKey = Bitconverter.touint32 (NewKetamahash (). ComputeHash (Namebytes),0); * Hostkeys.add (hashkey); $                     if(Hostdictionary.containskey (hashkey))Panax Notoginseng                     { -                         Throw NewException ("find the same hash key when creating the virtual node, check that the same server is passed in"); the                     } + Hostdictionary.add (HashKey, realhostarr[i]); A                 } the             } +  - Hostkeys.sort (); $Ketamahashkeys =Hostkeys.toarray (); $}

Distributive rules of consistent hash algorithm

Now that we know the hash value of all the virtual nodes, let's look at how we can get to the server when we have an object, or how to take out the object when we get the key to the object.

Set an object, the object's key first as the "Ketama" algorithm key, after calculating the hash value we need to do the following several steps.

1: First check whether the virtual node is equal to the current object hash value, if there is the object directly into the hash value of the same node, the subsequent steps will not continue.

2: If not, find the first node that is larger than the hash value of the current object, (the hash value of the node is sorted in ascending order, the ring corresponds clockwise), that is, the node closest to the object, and then the object is stored in that node.

3: If the hash value of the server is not found larger than the object, it proves that the hash value of the object is between the last node and the first node, that is, between E and a on the ring. This situation directly deposits the object into the first node, a.

Code implementation:

1 /// <summary>2         ///obtain the corresponding real server based on the hash key3         /// </summary>4         /// <param name= "hash" ></param>5         /// <returns></returns>6          Public stringGethostbyhashkey (stringkey)7         {8             byte[] bytes =Encoding.UTF8.GetBytes (key);9             UINThash = Bitconverter.touint32 (NewKetamahash (). ComputeHash (bytes),0);Ten  One             //look for the server that is equal to the current hash value. A             inti =Array.BinarySearch (Ketamahashkeys, hash); -  -             //If I is less than 0, it means that there are no virtual nodes equal to the hash value the             if(I <0) -             { -                 //the I continues the bitwise complement to get the first virtual node in the array that is greater than the current hash value -i = ~i; +  -                 //if the bitwise-Complement I is greater than or equal to the size of the array, then there is no virtual node in the array that is greater than the current hash value +                 //take the first server directly at this time A                 if(I >=ketamahashkeys.length) at                 { -i =0; -                 } -             } -  -             //returns the corresponding real server host address based on the hash key of the virtual node in             returnHostdictionary[ketamahashkeys[i]]; -}

Get an object, also through the "Ketama" algorithm to calculate the hash value, and then the same as the set process to find the node, found after the object can be directly removed.

So what does this "Ketama" look like, let's see the code implementation.

1 /// <summary>2     ///Ketama Hash Encryption algorithm3     ///about HashAlgorithm See MSDN links4     ///  http://msdn.microsoft.com/zh-cn/library/system.security.cryptography.hashalgorithm%28v=vs.110%29.aspx5     /// </summary>6      Public classKetamahash:hashalgorithm7     {8 9         Private Static ReadOnly UINTFnv_prime =16777619;Ten         Private Static ReadOnly UINTOffset_basis =2166136261; One  A         protected UINTHash; -  -          PublicKetamahash () the         { -Hashsizevalue = +; -         } -  +          Public Override voidInitialize () -         { +hash =offset_basis; A         } at  -         protected Override voidHashcore (byte[] Array,intIbstart,intcbsize) -         { -             intLength = Ibstart +cbsize; -              for(inti = Ibstart; i < length; i++) -             { inhash = (Hash * fnv_prime) ^Array[i]; -             } to         } +  -         protected Override byte[] hashfinal () the         { *Hash + = Hash << -; $Hash ^= Hash >>7;Panax NotoginsengHash + = Hash <<3; -Hash ^= Hash >> -; theHash + = Hash <<5; +             returnbitconverter.getbytes (hash); A         } the}

Test performance

Finally, I refer to my reference beitmemcached written algorithm and the old generation (discuz! De Shenjun) Reference spymemcached wrote a comparison.

The source code is downloaded later.

The result: the time to find 5W keys is 100 times faster than the older version, but it's a bit worse in terms of load balancing.

Test data:

1: The real server is 5 units

2: Randomly generated 5W string key (generation method takes the old generation directly)

3: The virtual nodes are all 250

My version:

Version of the old generation:

Resources

Beitmemcached Source

Old generation: C # Implementation of the consistent hash algorithm (Ketamahash)

Summarize the consistency hash (consistent Hashing)

Application of the consistent hash algorithm in memcached