Redis pre-Partitioning technology and implementation

In general, we will create multiple redis instances to relieve the pressure on a single redis instance. However, as the number of caches increases, resizing redis is a non-essential task. There are multiple ways to resize redis, such as increasing the maximum memory of each redis instance. This is only one of the solutions, and lacks flexibility and scalability. On the official redis website, we mentioned the pre-sharding technology. This article will explain the pre-sharding technology and explain how Jedis supports sharding.

I. redis pre-Partitioning technology

Build multiple redis instances on a single server. You can use the replication mechanism to expand an extension as follows:

1) Create an empty redis instance on the new server.

2) configure a new redis instance as the slave instance of the source instance and direct the source data to the new instance.

3) Stop the client (such as Jedis ).

4) Update the instance IP address configured on the client to the new server IP address. Note that the old IP address cannot be replaced.

5) Send the slaveof no one command on the new server so that it is no longer used as a slave instance.

6) restart the client with the new configuration.

7) Stop the old instances that are no longer used on the old server.

A question is raised here. After the old IP address is replaced, can the Keywords previously mapped to the old redis instance be mapped to the corresponding new redis instance? This article will be analyzed later.

II. Implementation of the sharding Technology in Jedis

Jedis supports the sharding technology. The main classes involved are divided into two parts:

1) record the address information of each redis instance. The class diagram shows that shardinfo and jedisshardinfo implement this function;

2) use consistent hash algorithm to map keywords to redis instances. From the class diagram, we can see that sharded provides a base class for this function. binaryshardedjedis and shardedjedis are byte and string implementations respectively.

An initialization method is executed when the sharded class is created. The hash algorithm generates 160 hash values for each apsaradb for redis instance, uses the hash value as the key of the treemap, and uses the shardinfo information of the apsaradb for redis instance as the value. Note: When calculating the hash value, it is not the IP address used, but an alias. This alias is either set in shardinfo or generated based on certain rules. For details, refer to the following code:

    private void initialize(List<S> shards) {nodes = new TreeMap<Long, S>();for (int i = 0; i != shards.size(); ++i) {    final S shardInfo = shards.get(i);    if (shardInfo.getName() == null)for (int n = 0; n < 160 * shardInfo.getWeight(); n++) {    nodes.put(this.algo.hash("SHARD-" + i + "-NODE-" + n),    shardInfo);}    elsefor (int n = 0; n < 160 * shardInfo.getWeight(); n++) {    nodes.put(    this.algo.hash(shardInfo.getName() + "*"    + shardInfo.getWeight() + n), shardInfo);}    resources.put(shardInfo, shardInfo.createResource());}    }

If you do not use the IP address for hash, the data in the cache cannot be accessed when the IP address is changed. I have answered the previous questions.

In fact, this is the implementation of the consistent hash algorithm. Memcached uses this algorithm. The 160 hash values are actually virtual nodes of a single instance in the hash ring. If you do not create a virtual node, it will cause pressure on a node when you increase or delete the node. If you create a virtual node, the pressure can be divided into various redis instances.

Zookeeper

Redis pre-Partitioning technology and implementation