The principle of consistent hashing algorithm and its application in Distributed system

This paper will introduce the consistent hashing algorithm (consistent Hashing) and its application in distributed system from the practical application scenario. First of all, this article will describe a problem scenario that is often encountered in daily development, by introducing a consistent hashing algorithm and how the algorithm solves the problem, and then describes the algorithm in a relatively detailed way, and discusses some topics, such as virtual nodes, related to the application of this algorithm. Distributed Cache issues

Suppose we have a website, recently found that as traffic increases, the server pressure is increasing, the way to read and write the database directly is not very good, so we want to introduce memcached as a caching mechanism. Now we have three machines available as memcached servers, as shown in.

Obviously, the simplest strategy is to randomly send each memcached request to a memcached server, but this strategy can pose two problems: one is that the same data may be present on different machines, resulting in data redundancy, and second, it is possible that some data has been cached but access is not hit , because all access to the same key is not guaranteed to be sent to the same server. Therefore, the stochastic strategy is very bad both in terms of time efficiency and space efficiency.

To solve these problems, just do the following: Ensure that access to the same key is sent to the same server. There are many ways to do this, and the most common method is to compute the hash. For example, for each access, the hash value can be computed as follows:

h = Hash (key)% 3

Where hash is a hash mapping function from string to positive integer. Thus, if we have memcached server numbered 0, 1, 2, then we can calculate the server number H according to the above and key, then go to access.

Although this approach solves the two problems mentioned above, there are some other problems. If the above method is abstracted, it can be considered by:

h = Hash (key)% N

This calculation calculates which server the request for each key should be sent to, where N is the number of servers, and the server is numbered by 0– (N-1).

The problem with this algorithm is that fault tolerance and extensibility are not good. Fault tolerance refers to whether the whole system can run correctly and efficiently when one or several servers in the system become unavailable, and extensibility refers to whether the whole system can run correctly and efficiently when a new server is added.

Now assume that a server is down, in order to fill the vacancy, to remove the outage server from the numbered list, the following server in order to move forward one and the number of the value minus one, at which point each key will be the H = Hash (key)% (N-1) recalculation; Similarly, if a new server is added, Although the original server number does not have to be changed, the hash value is recalculated by H = hash (key)% (n+1). Therefore, once there is a server change in the system, a large number of keys will be relocated to different servers resulting in a large number of cache misses. This is a very bad situation in a distributed system.

A well-designed distributed hashing scheme should have good monotonicity, that is, the increase or decrease of service nodes will not cause a lot of hash relocation. A consistent hashing algorithm is one such hash scheme.

A brief description of consistency hashing algorithm algorithm

The consistent hashing algorithm (consistent Hashing) was first published in the paper consistent Hashing and Random trees:distributed Caching protocols for relieving hot Spot s on the World Wide Web. In simple terms, a consistent hash organizes the entire hash value space into a virtual ring, such as assuming that the value space of a hash function h is 0-232-1 (that is, the hash value is a 32-bit unsigned shape), and the entire hash space loop is as follows:

The entire space is organized in a clockwise direction. 0 and 232-1 coincide in the direction of 0 points.

The next step is to use H to make a hash of each server, select the server's IP or hostname as the keyword hash, so that each machine can determine its location on the Hashi, this assumes that the above three servers using the IP address hash after the location of the ring space is as follows:

Next, use the following algorithm to locate the data access to the appropriate server: The data key using the same function H to calculate the hash value H, by the H to determine the position of this data on the ring, from this position along the ring clockwise "walk", the first server encountered is the server it should be located.

For example, we have a, B, C, D four data objects, after hashing, the position on the ring space is as follows:

Based on the consistent hashing algorithm, data A is set to server 1, D is set to server 3, and B and C are set to server 2, respectively.

Fault Tolerance and Scalability analysis

The following is an analysis of the fault tolerance and extensibility of the consistent hashing algorithm. Now assume that server 3 is down:

You can see that a, C, B are not affected at this point, only the D node is relocated to server 2. In general, in a consistent hashing algorithm, if a server is unavailable, the affected data is only data between the server and the previous server in its ring space (that is, the first server encountered in the counterclockwise direction), and the others are unaffected.

Consider the other case if we add a server to the system memcached Server 4:

At this point A, D, C are unaffected and only B needs to be relocated to the new server 4. In general, in a consistent hashing algorithm, if you add a server, the affected data is only the data between the new server and the previous server in its ring space (that is, the first server encountered in the counterclockwise direction), and the others are unaffected.

To sum up, the consistency hashing algorithm can only reposition a small subset of data in the ring space for the increment and decrease of the nodes, which has good fault tolerance and expansibility.

Virtual node

The consistency hashing algorithm is too young for the service node, and is prone to data skew due to uneven node division. For example, there are two servers in our system, and their rings are distributed as follows:

This inevitably results in a large amount of data being concentrated on server 1, and only a very small number will be located on server 2. In order to solve this data skew problem, the consistent hashing algorithm introduces the virtual node mechanism, that is, to compute multiple hashes for each service node, and to place a service node, called a virtual node, for each computed result location. This can be done by adding numbers to the server IP or host name. For example, we decided to compute three virtual nodes for each server, so we can calculate "Memcached server 1#1", "Memcached server 1#2", "Memcached server 1#3", "Memcached Server 2#1 "," Memcached server 2#2 "," Memcached server 2#3 "hash values, resulting in six virtual nodes:

At the same time, the data location algorithm is not changed, just one step more virtual node to the actual node mapping, such as positioning to "Memcached server 1#1", "Memcached server 1#2", "Memcached server 1#3" Data for three virtual nodes is located on server 1. This solves the problem of data skew when the service node is young. In practical applications, the number of virtual nodes is usually set to 32 or greater, so even a few service nodes can achieve a relatively uniform data distribution.

Summarize

The current consistent hash is basically a standard configuration for distributed system components, such as Memcached's various clients that provide built-in consistent hash support. This article is just a brief introduction to this algorithm, more in-depth content can be see the paper "Consistent Hashing and Random trees:distributed Caching protocols for relieving hot spots on t He world Wide Web, and also provides a C language version of the implementation for reference.

Ext.: http://blog.codinglabs.org/articles/consistent-hashing.html

===================================

Two. Algorithm design 1. Source of the problem

A service consisting of 6 servers, each server responsible for storing 1/6 of the data, and when the service is re-usable when the Server1 is down.

As can be clearly seen in the table below, when the Server1 outage, HASH1 service is completely unavailable, so need to rehash by the remaining 5 machines to provide all the data services, but because each machine responsible for the size of the data segment is not the same, you need to migrate data between different servers, And the service is unavailable until the data migration is complete.

2. Classic Consistent hashing algorithm

Aiming at the disadvantage of rehash, Karger proposes an algorithm, the core of which is "virtual node".

Map all the data into a set of virtual nodes that are larger than the number of servers, and then map the virtual nodes to the real servers. So when the server goes down, because the number of virtual nodes is fixed, all do not need to rehash, and only need to re-migrate the virtual nodes that the service is not available, so that only the data of the outage node needs to be migrated.

In the classic algorithm, the next real node of the outage server will provide the service.

Three. Algorithm improvement 1. Problems with the classic consistent hashing algorithm

The classic algorithm only solves the flaw of the rehash algorithm, when itself is not perfect. There are several main issues:

(1) Server1 downtime leads to Server2 service being subjected to one-fold data service, and if Server1 is retired, the overall system load is completely unbalanced.

(2) If all the server can withstand the data read and write, then if under normal circumstances all the data write two copies to different servers, the main standby or load balancer, down to read the backup node data directly, there is no need to appear in the classic algorithm data migration.

2.Dynamo Improvement Practices

Amazon's big Data storage platform "Dynamo" uses a consistent hash, but it does not use the classic algorithm, but instead uses the idea of a failed node rehash.

The system saves the correspondence of all virtual nodes and real servers to a configuration system, and when the services of some virtual nodes are unavailable, reconfigure the services of these virtual nodes to other real servers, so as not to migrate the data in large quantities, and to ensure that the load of all servers is relatively balanced.

Virtual node	0-4/5	10-14/6	15-19/7	20-24/8	24-29/9
Recovery	Server0	Server2	Server3	Server4	Server5

Four. Algorithm extension

The consistency hashing algorithm is used to solve the problem of server outage and capacity expansion, but the idea of "virtual node" has been developed, and some distributed systems are used to realize load balancing and optimal access strategy of the system.

In a real system scenario, two systems with the same deployment may not provide the same service, mainly because:

(1) The individual hardware differences result in different server performance.

(2) The network communication efficiency between IDC servers is different because of the computer room switch and network bandwidth.

(3) The service performance of telecom IDC and China Unicom IDC can be different with different network operators.

(4) The server is in the network or computer room encountered an attack.

So the exact same two sets of systems may also need to provide differentiated services, through the use of virtual nodes can be flexibly adjusted dynamically to achieve the optimization of system services.

For a distributed system consisting of 2 nodes and 3 servers per node, s0-1 is the SERVER0 of Distributed System 1, the system configuration administrator can dynamically adjust the mapping relationship between virtual node and real server according to the real service efficiency of the system. It is also possible for the customer system to adjust its own access policy depending on the response rate or response time.

Reproduced in original: http://blog.csdn.net/bzhxuexi/article/details/46422027

The principle of consistent hashing algorithm and its application in Distributed system