Redis replication and scalable cluster setup

This article discusses the replication capabilities of Redis and the pros and cons of the Redis replication mechanism itself, as well as cluster setup issues.

Overview of the Redis replication process

The Redis replication feature is based on a memory-snapshot-based persistence strategy that we discussed earlier, which means that no matter what your persistence strategy chooses, if you use the Redis replication feature, there will be a memory snapshot, so first pay attention to your system memory capacity planning, The reason may refer to the Redis disk IO issue mentioned in my previous article.

The Redis replication process is a set of state machine flows on the slave and Master side, and the state information involved is:

Slave End:

redis_repl_noneredis_repl_connectredis_repl_connected

Master side:

Redis_repl_wait_bgsave_startredis_repl_wait_bgsave_endredis_repl_send_bulkredis_repl_online

The entire state machine process process is as follows:

1. The slave side adds the slave of instruction in the configuration file, so slave reads the configuration file at startup and the initial state is redis_repl_connect.

2. The slave end connects master in the timed task Servercron (Timer trigger event within Redis), sends the Sync command, and then blocks waiting for master to send back its memory snapshot files (the latest version of Redis has no need to let slave block).

3. The master side receives the sync command to simply determine if there is an ongoing memory snapshot subprocess, does not immediately start a memory snapshot, and then waits for it to end, sending the file to the slave end when the snapshot is complete.

4. The slave side receives the memory snapshot file from master, saves it locally, clears the memory table when it is received, re-reads the memory snapshot file from master, rebuilds the entire memory table data structure, and the final state is set to the redis_repl_connected state. Slave state machine flow is complete.

5. The master side in the process of sending the snapshot file, any command that changes the data set will be temporarily saved in the slave network connection of the Send cache queue joins (list data structure), after the snapshot is completed, sent to slave, followed by the same processing of the command received, and the status is set to Redis_ Repl_online.

The entire replication process is complete, as shown in the process:

Flaws in the Redis replication mechanism

As can be seen from the above process, slave from the library when connected to the Master Master Library, Master will take a memory snapshot, and then send the entire snapshot file to slave, that is, there is no replication location like MySQL concept, that is, no incremental replication, which will bring a lot of problems for the entire cluster construction.

For example, a running master library on a line is configured with a simple read-write separation from the library, when slave is disconnected from master due to network or other reasons, then when slave is reconnected, it needs to regain the memory snapshot of the whole master. Slave all the data and then re-establish the entire memory table, on the one hand slave recovery time will be very slow, on the other hand will also bring pressure to the main library.

So for these reasons, if your Redis cluster requires master-slave replication, then it is best to configure all the slaves in advance to avoid adding from the library.

Cache or storage?

After we have analyzed the replication and persistence capabilities of Redis, it is not difficult to come to a conclusion that the current version of Redis is still a stand-alone version of the idea, the main problem is that the persistence is not mature enough, the replication mechanism has a relatively large defect, Then we started rethinking the location of Redis: Cache or storage?

If, as a cache, it seems that in addition to some very special business scenarios, it is necessary to use some kind of REDIS data structure, we may be more suitable to use memcached, after all, memcached regardless of the client package and the server itself more proven.

If it is a storage storage, the biggest problem we face is whether it is persistent or replication has no way to solve the Redis single point problem, that is, a redis hangs, there is no good way to recover quickly, usually dozens of g of persistent data, Redis Restart loading takes a few hours, and replication is flawed, how to solve it?

Redis Scalable Cluster Setup1. Proactive replication avoids redis replication defects.

Since the replication capabilities of Redis are flawed, we might as well abandon the replication capabilities provided by Redis itself, and we can use proactive replication to build our cluster environment.

The so-called active replication refers to the data of Redis stored by the business side or through the agent middleware to write or write, the data of the multiple storage to achieve the same purposes as replication, active replication is not limited to the use of Redis cluster, Many companies now employ proactive replication techniques to address the delay between MySQL master and slave replication, such as Twitter, which has also developed middleware gizzard for replication and partitioning (Https://github.com/twitter/gizzard ).

Although active replication solves the problem of passive replication delay, it also brings new problems, that is, the consistency of data, data write 2 or more times, how to ensure the consistency of multiple data? If your application does not require high data consistency and allows for eventual consistency, then the usual simple solution is to allow the client to simultaneously fetch multiple copies of the data and verify it by using a timestamp or a vector clock, if your application is highly consistent with the data, It is necessary to introduce some complex consistency algorithms such as Paxos to ensure the consistency of the data, but the write performance will be reduced a lot correspondingly.

With active replication, multiple storage of data we are no longer concerned about Redis single point of failure, and if a group of redis clusters hangs up, we can quickly switch the business to another set of redis, reducing business risk.

2. Redis online expansion via presharding.

Through active replication we solve the problem of Redis single point of failure, then there is an important problem to solve: capacity planning and online expansion.

The scenario in which we previously analyzed Redis is that all of the data is stored in memory and memory capacity is limited, so you first need to do initial capacity planning based on the amount of business data, such as your business data requires 100G of storage space, assuming that server memory is 48G, So, according to the previous issue of redis disk IO, we need about a few or two servers to store it. This is actually a capacity planning for the existing business situation, if the business grows quickly, will soon find that the current capacity is not enough, redis stored in the data will soon exceed the physical memory size, then how to expand the online Redis?

The Redis author proposes a scheme called presharding to solve the problem of dynamic expansion and data partitioning, which is actually the way to deploy multiple Redis instances on the same machine, when there is not enough capacity to split multiple instances onto different machines, which actually achieves the effect of scaling.

The split process is as follows:

1. Start a Redis instance of the corresponding port on the new machine.

2. Configure the new port to be the slave library for the port to be migrated.

3. After the replication is complete, after synchronizing with the main library, switch all the client configurations to the new port from the library.

4. Configure the new main library from the library.

5. Remove the old Port instance.

6. Repeat the above procedure to migrate all ports to the specified server.

The above split process is a smooth migration by the Redis author, but the split method relies heavily on the replication capabilities of Redis itself, and if the main library snapshot data file is too large, this replication process will be too long and will put pressure on the main library. So the process of doing this split is best chosen for the low peak hours of business access.

Improved approach to Redis replication

Our online system uses our own improved version of Redis, which mainly solves the bug that Redis does not have incremental replication, and is able to complete a mysql-like binlog that can be incrementally replicated by requesting log locations from the library.

Our persistence scheme is to first write Redis's aof file, and automatically split the aof file by file size, close the Redis rewrite command, and then store the memory snapshot at the low peak time of the business and write the current aof file location to the snapshot file. So we can make the snapshot file and the location of the aof file consistent, so that we have a memory snapshot of the system at some point, and also know the location of the corresponding aof file at this moment, when the synchronization command is sent from the library, we first send the snapshot file to the slave library, and then from the library to take The location of the aof file stored in the snapshot file, and the location is sent to the main library, which then sends all commands after that location, and the subsequent replication is the incremental information after that location.

The combination of Redis and MySQL

At present, most Internet companies use MySQL as the main persistent storage of data, so how to make Redis and MySQL well together? We mainly use a MySQL as the main library, Redis as a high-speed data query from the library of heterogeneous read-write separation scheme.

For this purpose, we have developed our own MySQL replication tool, which makes it easy to synchronize data from MySQL to Redis in real time.

(Mysql-redis heterogeneous read-write separation)

Summarize:

1. The Redis replication feature does not have incremental replication, and each time the re-connection sends the entire memory snapshot of the main library to the slave library, you need to avoid adding from the library to the more stressful main libraries on the online service.

2. Redis replication is using snapshot persistence, so if your redis persistence option is log append (AOF), then it is possible for the system to do both the AOF log file and the snapshot write disk operation at the same time. The responsiveness of Redis will be affected at this time. So if you choose aof Persistence, you need to be more cautious about adding from the library.

3. Redis cluster construction and online expansion can be done using active replication and presharding methods.

Redis replication and scalable cluster setup