CARP principle analysis of consistent hashing algorithm with Golang implementation

This is a creation in Article, where the information may have evolved or changed.

CARP principle analysis of consistent hashing algorithm with Golang implementation

In the process of backend service development, there is a problem: you need to deploy Redis in front of MySQL to do a layer of cache, requires Redis is a cluster deployment, and each Redis node only cache the total amount of data 1/n, N is the number of Redis.

See here everyone can think of a method is used hash(key)%N to select Redis for value access, this way can be very evenly distributed data to the N Redis service, and implementation is very simple. But there is a big problem with the way this hash is used, is when the Redis cluster expands or shrinks, or when the outage occurs, that is, the above formula N changes, this time hash(key)%N the value remains the same probability is very small, in other words, the cache system in this case the entire failure, which for the back end of the MySQL transient pressure will be very large , it is likely to cause MySQL to collapse, which in turn will make the entire service unusable.

So you have to think of a way to deal with the above situation, that is, when a Redis service hangs up, can you do only 1/n cache invalidation?

The answer is to use the consistent hash algorithm CARP, strictly speaking CARP is not an algorithm, but a protocol, the Cache Array Routing protocol,cache Group Routing protocol. Here's how it works:

First assume that there are N redis services, REDIS1, Redis2 ... redisn, key is the one you want to get the data in Redis.

The first step is to calculate the hash value of all service names and keys:

hash_v1 = hash(redis1 + key)hash_v2 = hash(redis2 + key)...hash_vN = hash(redisN + key)

The second step is to calculate the maximum hash_vx, then the REDISX is selected:

hash_vX = max(hash_v1, hash_v2, ..., hash_vN)

Why is it possible to do this by adding a Redis service when only the 1/(n+1) cache fails?

hash_vX1 = max(hash_v1, hash_v2, ..., hash_vN)hash_vX2 = max(hash_v1, hash_v2, ..., hash_vN, hash_vN+1)

Look at the above two expressions, if the requirement is hash_vX2 greater than hash_vX1 the hash_vN+1 previous n hashes, then the hash function you choose is sufficient to hash, hash_vN+1 the probability is greater than the previous N hash value is 1/(n+1). You can do it yourself. Reduce one R The probability that the cache fails when Edis service.

Finally, attach the Golang implementation version:

package carpimport (    "crypto/md5"    "errors")var (    ErrHaveNoRedis = errors.New("have no redis"))// 根据 carp 算法, 从 redisArr 的数组中选择合适的 redis, 返回值为下标func GetTargetIndex(key string, redisArr []string) (idx int, err error) {    if len(redisArr) < 1 {        return -1, ErrHaveNoTarget    } else if len(redisArr) == 1 {        return 0, nil    }    hashArr := make([]string, len(redisArr))    for k, v := range redisArr {        hashArr[k] = hashString(v + key)    }    idx = minIdx(hashArr)    return}// 返回 arr 数组中最小值的下标func minIdx(arr []string) (idx int) {    if len(arr) < 1 {        return -1    }    idx, min := 0, arr[0]    for k, v := range arr {        if v < min {            idx = k            min = v        }    }    return}func hashString(str string) (hash string) {    md5sum := md5.Sum([]byte(str))    return string(md5sum[:])}

The MD5 hash algorithm is used in the above code, and if you have a higher performance requirement you can use the FNV hashing algorithm.

Finally, you can also check out this article on my own blog:

http://tips.codekiller.cn/201 ...