Hash ring/consistent hashing consistent hashing algorithm

Tag: It's stored in different IP address relationships. Three many illustrations csdn

A distributed hash (DHT) implementation algorithm, proposed by MIT in 1997, was designed to address hot spot problems in the Internet, with a similar intent to carp.      The consistent hash corrects the problem caused by the simple hashing algorithm used by carp, so that distributed hashing (DHT) can be really applied in the peer-to-peer environment. The consistency hash algorithm proposes four definitions for determining the good or bad hash algorithm in a dynamically changing cache environment:

1, Balance (Balance): The balance is that the result of the hash can be distributed as far as possible in all buffers, so that all buffer space can be exploited. Many hashing algorithms can satisfy this condition.

2. Monotonicity (monotonicity): monotonicity means that if some content has been allocated to the corresponding buffer by hashing, a new buffer is added to the system.  The result of the hash should be to ensure that the original allocated content can be mapped to an existing or new buffer without being mapped to another buffer in the old buffer collection.

3, Dispersion (Spread): In a distributed environment, the terminal may not see all the buffers, but only to see part of it. The end result is that the same content is mapped to different buffers by different endpoints when the terminal wants the content to be mapped to buffering through a hashing process, because the buffer range seen by different terminals may be different, resulting in inconsistent results for the hash. This is obviously something that should be avoided because it causes the same content to be stored in different buffers, reducing the efficiency of the system's storage. The definition of dispersion is the severity of the above-mentioned situation.  A good hashing algorithm should be able to avoid inconsistencies as far as possible, that is, to minimize dispersion.

4. Load: The load problem is actually looking at the dispersion problem from another perspective. Since different terminals may map the same content to different buffers, it is possible for a particular buffer to be mapped to different content by different users. As with dispersion, this situation should also be avoided, so a good hashing algorithm should be able to minimize the buffering load.

     In a distributed cluster, adding or removing machines, or automatically leaving the cluster after a machine failure, is the most basic function of distributed cluster management. If the use of commonly used hash (object)%n algorithm, then after the machine is added or deleted, many of the original data can not be found, which seriously violates the monotony principle. Next, the main explanation is how the consistent hashing algorithm is designed:  ring hash space According to the commonly used hash algorithm to hash the corresponding key into a space with a 2^32 bucket, namely 0~ (2^32)-1 of the digital space. Now we can connect the numbers to each other and think of them as a closed loop. e.g.                                   &N Bsp                          ,         &NB Sp   mapping data through a certain hash algorithm to the ring now we will Object1, Object2, Object3, Object4 four objects through a specific hash function to calculate the corresponding key value, and then hash to the hash ring. such as:    hash (object1) = key1;    hash (object2) = key2;    hash (OBJECT3) = key3;    H Ash (OBJECT4) = key4;                            &NBS P                               The machine is mapped to the ring over a hash algorithm A distributed cluster with a consistent hashing algorithm will use the new machineIn addition, the principle is that by using the same hash algorithm as object storage, the machine is mapped to the ring (in general, the machine's hash is calculated using the machine's IP or machine's unique alias as the input value), and then in the clockwise direction, all objects are stored in the machine closest to their own. Suppose now have node1,node2,node3 three machines, through the Hash algorithm to get the corresponding key value, mapped to the ring, which is as follows: hash (NODE1) = KEY1; Hash (NODE2) = KEY2; Hash (NODE3) = key3;                          &NBSP ;                                 to see Objects and machines In the same hash space, so that clockwise rotation object1 stored in the NODE1, object3 stored in NODE2, object2, Object4 stored in NODE3. In such a deployment environment, the hash ring is not changed, so, by calculating the object's hash value can be quickly positioned to the corresponding machine, so that the object can find the real storage location.   The removal of the machine and the addition of ordinary hash algorithm is the most inappropriate place is in the addition or deletion of the machine will be taken as a large number of object storage location failure, so it is not satisfied with the monotony of the big. The following is an analysis of how the consistent hashing algorithm is handled. 1. Node (machine) removal     with the above distribution as an example, if the NODE2 failure is removed, then according to the method of clockwise migration, OBJECT3 will be migrated to NODE3, so that only the OBJECT3 mapping location has changed, There are no changes to the other objects. :                                  nbsp                   &nbsp       2. Node (machine) Add      If you add a new node NODE4 to the cluster, you get KEY4 through the corresponding hash algorithm and map to the ring, such as:            & nbsp                          ,         &NB Sp                 by moving the rules clockwise, the Object2 is migrated to NODE4, and the other objects maintain the original storage location. Through the analysis of the addition and deletion of the nodes, the consistency hashing algorithm keeps the monotonicity while the data is migrated to a minimum, so the algorithm is suitable for the distributed cluster, avoids the large amount of data migration, and reduces the pressure of the server.   Balance According to the above graphic analysis, the consistent hashing algorithm satisfies the characteristics of monotonicity and load balancing and the dispersion of the general hash algorithm, but it is not considered as a widely used origin, because it lacks the balance. The following will analyze how the consistent hashing algorithm is balanced. Hash algorithms are not guaranteed to be balanced, such as the case where only NODE1 and NODE3 are deployed (NODE2 deleted), object1 are stored in NODE1, Object2, OBJECT3, object4 are stored in NODE3, This is a very unbalanced state. In the consistent hashing algorithm, the virtual node is introduced in order to satisfy the balance as much as possible.    --"Virtual node" is the actual node (machine) in the hash space of the replica (replica), a real node (machine) corresponding to a number of "virtual node", the corresponding number has become "Replication Number", "Virtual node" in Hash spaces are listed as hash values. As an example of the above only deployed NODE1 and NODE3 (NODE2 deleted diagram), the previous objects are unevenly distributed on the machine, now we take 2 copies (copy number) as an example, so that there are 4 virtual nodes in the entire hash ring, and the graph of the final object mapping is as follows:       &NBSP                          ,         &NB Sp                       based on the mapping relationship of:object1->node1-1,object2-> Node1-2,object3->node3-2,object4->node3-1. Through the introduction of virtual nodes, the distribution of objects is more balanced. So how does a real object query work in real-world operations? Conversion of objects from hash to virtual node to actual node such as:                            &N Bsp The hash calculation of             "Virtual node" can be based on the IP address of the corresponding node plus the digital suffix. For example, assume that the IP address of the NODE1 is 192.168.1.100. Before introducing "Virtual node", calculate the hash value of cache A: hash ("192.168.1.100"), after introducing "virtual node", calculate the hash value of "virtual section" point Node1-1 and Node1-2: hash ("192.168.1.100#1") ; Node1-1hash ("192.168.1.100#2"); Node1-2

Reprinted from: http://blog.csdn.net/cywosp/article/details/23397179

Hash ring/consistent hashing consistent hashing algorithm