Design and implementation of Codis part 2

This is a creation in Article, where the information may have evolved or changed.

Multi-Proxy architecture

In the design of Codis, Proxy is designed to be stateless, and the client connects any one proxy is the same. And each Proxy starts, it registers a temporary node on the Zookeeper, so the client can even implement HA based on this feature (actually I wrote a Jedis-based Codis ha roundrobinpool inside the pea pod)

The benefit of this design, of course, is that requests can be load balanced and there is no single point in the entire system. However, the problem comes, because Codis is the function of dynamic scaling capacity, when Codis in the process of data migration, how to ensure that any one Proxy will not read the old or wrong data?

Before explaining this question, I would like to introduce the data storage method of Codis and some pre-knowledge about data migration:

• Data is based on key, distributed in 1024 slots, the slot is a virtual concept, the data is stored in the actual multiple Codis-server (Codis modified version of Redis-server), each codis-server responsible for a part key-val UE data, hash algorithm is CRC32 (key)% 1024
• Data migration is initiated by Codis-config, and in Codis-config's view, the smallest unit of data migration is the slot
• For Codis-server, there is no distributed logic in it, just a few instructions for data transfer: Slotsmgrtone, Slotsmgrt .... Its main function is: Randomly select a specific slot in a Key-value pair, transmitted to another codis-server, the transmission succeeds, the local key-value pair Delete, note that the entire operation is atomic.

So, this determines that the Codis is not very suitable for key, but the value of the application is very large, and your key less, the greater the value, the end will degenerate into a single Redis model (performance is not as good as raw Redis), so Codis more suitable for mass key , the value is relatively small (<= 1 MB) of the app.

Why Codis-server data migration is a key, not similar to many other distributed systems, the use of replication way? I think, for the Redis system, the implementation of replication is not very economical, first of all, you need an RDB dump it? In Redis all the operations are strictly serial (single-threaded model decision), so dump data is required to fork a new process, otherwise if the direct SAVE will block the only main thread, but also to consider the dump process and the transfer process of the new data generated in the synchronization of the problem, More complex to implement. So each time we only move a key, not the main thread block, Redis operation is memory, batch write-once and multiple sets almost no difference (for single machine), and this model also avoids the migration process of data update synchronization problem, because the migration of a key Operation is atomic, and for this redis-server, it will not respond to other requests until the migration instruction is completed. So the security of the data is ensured.

A typical migration process:

1. Codis-config initiating migration instructions such as Pre_migrate slot_1 to group 2

2. Codis-config wait for all the proxy reply received the migration instructions, if a proxy does not respond, mark its downline (because the proxy starts on the ZK register a temporary node, if the proxy hangs, normally, this temporary node will also be deleted, in Codis -config found no response, Codis-config will wait for 30s, waiting for its offline, if not offline or still not responding, then Codis-config will not release the lock, notify the administrator of the problem is equivalent to a 2-phase commit

3. The status of the Codis-config tag slot_1 is migrate, the server group serving the slot is changed to Group2, and Codis-config is constantly being sent group1 commands to the SLOTSMGRT Redis machine. The target parameter is the group2 machine until there is no remaining key in the group1 that belongs to Slot_1

4. During the migration process, if the client requests slot_1 key data, proxy forwards the request to group2, and the proxy will first forcibly execute the MIGRATE key on group1 to migrate the key value forward. And then go to the group2 and read it normally.

5. Migration complete, tagged slot_1 status is online

Key points:

1. All operation commands are routed through Zookeeper, and all routing tables are placed in Zookeeper, ensuring that the view of any proxy is the same.

2. Codis-config will ensure that all proxies receive this migration request before actually modifying the slot state.

3. Before the client reads data in the slot being migrated, it forces the operation of the source Redis to perform the migration of this key.

These two points guarantee that the proxy will always be able to hit this key on the migrated target when it reads the data.

This is how Codis carries out a secure data migration process.