Consistent hashing algorithm and its application in Distributed system

Original: http://blog.codinglabs.org/articles/consistent-hashing.html

This paper will introduce the consistent hashing algorithm (consistent Hashing) and its application in distributed system from the practical application scenario. First of all, this article will describe a problem scenario that is often encountered in daily development, by introducing a consistent hashing algorithm and how the algorithm solves the problem, and then describes the algorithm in a relatively detailed way, and discusses some topics, such as virtual nodes, related to the application of this algorithm.

Distributed Cache issues

Suppose we have a website, recently found that as traffic increases, the server pressure is increasing, the way to read and write the database directly is not very good, so we want to introduce memcached as a caching mechanism. Now we have three machines available as memcached servers, as shown in.

Obviously, the simplest strategy is to randomly send each memcached request to a memcached server, but this strategy can pose two problems: one is that the same data may be present on different machines, resulting in data redundancy, and second, it is possible that some data has been cached but access is not hit , because all access to the same key is not guaranteed to be sent to the same server. Therefore, the stochastic strategy is very bad both in terms of time efficiency and space efficiency.

To solve these problems, just do the following: Ensure that access to the same key is sent to the same server. There are many ways to do this, and the most common method is to compute the hash. For example, for each access, the hash value can be computed as follows:

h = Hash (key)% 3

Where hash is a hash mapping function from string to positive integer. Thus, if we have memcached server numbered 0, 1, 2, then we can calculate the server number H according to the above and key, then go to access.

Although this approach solves the two problems mentioned above, there are some other problems. If the above method is abstracted, it can be considered by:

h = Hash (key)% N

This calculation calculates which server the request for each key should be sent to, where N is the number of servers, and the server is numbered by 0– (N-1).

The problem with this algorithm is that fault tolerance and extensibility are not good. Fault tolerance refers to whether the whole system can run correctly and efficiently when one or several servers in the system become unavailable, and extensibility refers to whether the whole system can run correctly and efficiently when a new server is added.

Now assume that a server is down, in order to fill the vacancy, to remove the outage server from the numbered list, the following server in order to move forward one and the number of the value minus one, at which point each key will be the H = Hash (key)% (N-1) recalculation; Similarly, if a new server is added, Although the original server number does not have to be changed, the hash value is recalculated by H = hash (key)% (n+1). Therefore, once there is a server change in the system, a large number of keys will be relocated to different servers resulting in a large number of cache misses. This is a very bad situation in a distributed system.

A well-designed distributed hashing scheme should have good monotonicity, that is, the increase or decrease of service nodes will not cause a lot of hash relocation. A consistent hashing algorithm is one such hash scheme.

A brief description of consistency hashing algorithm algorithm

The consistent hashing algorithm (consistent Hashing) was first published in the paper consistent Hashing and Random trees:distributed Caching protocols for relieving hot Spot s on the World Wide Web. In simple terms, a consistent hash organizes the entire hash value space into a virtual ring, such as assuming that the value space of a hash function h is 0-232-1 (that is, the hash value is a 32-bit unsigned shape), and the entire hash space loop is as follows:

The entire space is organized in a clockwise direction. 0 and 232-1 coincide in the direction of 0 points.

The next step is to use H to make a hash of each server, select the server's IP or hostname as the keyword hash, so that each machine can determine its location on the Hashi, this assumes that the above three servers using the IP address hash after the location of the ring space is as follows:

Next, use the following algorithm to locate the data access to the appropriate server: The data key using the same function H to calculate the hash value H, by the H to determine the position of this data on the ring, from this position along the ring clockwise "walk", the first server encountered is the server it should be located.

For example, we have a, B, C, D four data objects, after hashing, the position on the ring space is as follows:

Based on the consistent hashing algorithm, data A is set to server 1, D is set to server 3, and B and C are set to server 2, respectively.

Fault Tolerance and Scalability analysis

The following is an analysis of the fault tolerance and extensibility of the consistent hashing algorithm. Now assume that server 3 is down:

You can see that a, C, B are not affected at this point, only the D node is relocated to server 2. In general, in a consistent hashing algorithm, if a server is unavailable, the affected data is only data between the server and the previous server in its ring space (that is, the first server encountered in the counterclockwise direction), and the others are unaffected.

Consider the other case if we add a server to the system memcached Server 4:

At this point A, D, C are unaffected and only B needs to be relocated to the new server 4. In general, in a consistent hashing algorithm, if you add a server, the affected data is only the data between the new server and the previous server in its ring space (that is, the first server encountered in the counterclockwise direction), and the others are unaffected.

To sum up, the consistency hashing algorithm can only reposition a small subset of data in the ring space for the increment and decrease of the nodes, which has good fault tolerance and expansibility.

Virtual node

The consistency hashing algorithm is too young for the service node, and is prone to data skew due to uneven node division. For example, there are two servers in our system, and their rings are distributed as follows:

This inevitably results in a large amount of data being concentrated on server 1, and only a very small number will be located on server 2. In order to solve this data skew problem, the consistent hashing algorithm introduces the virtual node mechanism, that is, to compute multiple hashes for each service node, and to place a service node, called a virtual node, for each computed result location. This can be done by adding numbers to the server IP or host name. For example, we decided to compute three virtual nodes for each server, so we can calculate "Memcached server 1#1", "Memcached server 1#2", "Memcached server 1#3", "Memcached Server 2#1 "," Memcached server 2#2 "," Memcached server 2#3 "hash values, resulting in six virtual nodes:

At the same time, the data location algorithm is not changed, just one step more virtual node to the actual node mapping, such as positioning to "Memcached server 1#1", "Memcached server 1#2", "Memcached server 1#3" Data for three virtual nodes is located on server 1. This solves the problem of data skew when the service node is young. In practical applications, the number of virtual nodes is usually set to 32 or greater, so even a few service nodes can achieve a relatively uniform data distribution.

Summarize

The current consistent hash is basically a standard configuration for distributed system components, such as Memcached's various clients that provide built-in consistent hash support. This article is just a brief introduction to this algorithm, more in-depth content can be see the paper "Consistent Hashing and Random trees:distributed Caching protocols for relieving hot spots on t He world Wide Web, and also provides a C language version of the implementation for reference.

Consistent hashing algorithm and its application in Distributed System (RPM)