Brief discussion on Redis and master copy business information

The senior research and development engineer of Jingdong Cloud Platform-Zhangchengyuan Redis is an open-source, Key/value caching system that adheres to the BSD License agreement and is renowned for its efficient response and rich data structures. The use of Redis in Jingdong is also very common, including the use of a lot of key business, because Redis official cluster has not been released, in the process of using redis need to face Redis single point of problem, Jingdong adopted a more general solution that is, the master-slave backup plus the corresponding master-slave switching (in some scenarios may be read and write separation), so that the main redis appear when the failure can be quickly switched to from the Redis. But one of the problems with Redis is that, in the event of network instability, slave and master disconnects (including Flash breaks) can cause master to regenerate all of the data in memory rdb files (snapshot files) and then transmit them to slave. After receiving the Rdb file from master, slave will empty its memory and reload the Rdb file into memory. This way inefficient, especially in the case of large amount of data, after all, the network flash is not necessarily lost data or lost data is only a small part, but to pay for the entire memory data are retransmitted once the price. If you can pass the update data for the flash break to slave, you do not need to pass all the data in master memory to slave. The Redis author has implemented this partial replication in the candidate version of 2.8 (hereinafter referred to as Redis2.8). So how is Redis2.4.16 's total copy and Redis2.8 partial replication implemented? As shown in the following illustration, these 5 states are slave in several states involved in the master-slave replication process, where Redis_repl_none is the default state for Redis startup. The four states shown in Figure 1-2 represent the state of slave in the view of master, because the slave is a special client on the master side (the same is true for the slave side). Figure 1-1 Slave own state diagram 1-2 the slave state of the master end Redis after receiving the "Slaveof IP Port" command, the state is first set to Redis_repl_connect, indicating that it needs to connect to its master. At this point slave did not connect to master. Redis calls the Servercron () function once every 100ms, and every 10 times Servercron () calls Replicationcron () once, and the replication () function is called once per 1s. InThe replication () function checks the state of the slave and, if it is in a redis_repl_connect state, creates an event handler for Syncwithmaster () and changes the slave state to Redis_repl_ Connecting. The Syncwithmaster () function mainly sends the sync command to master, and when the event handler is triggered, the slave state is changed to Redis_repl_transfer. Indicates that slave is ready to receive master-generated RDB files. Returning to Master's role, Master discovers that there is a slave connection, and if the master at this time does not have a slave and does not have a background snapshot process, start a background process to generate the current in-memory data into a RDB file while the slave state is set to Redis _repl_wait_bgsave_end state that represents the end of the snapshot process for the slave waiting for master. When a RDB file is generated in the background, if there is an update command for Redis, Master saves the update commands to the slave buffer and, if the buffer is full, opens up a separate list to store the update commands. When the background snapshot process finishes, master changes the state of the slave to Redis_repl_send_bulk and registers the Sendbulktoslave () event handler function to transfer the generated RDB file to slave. After the RDB transmission is over, the Sendbulktoslave () event function is deleted, the slave state is changed to Redis_repl_online, and the Sendreplytoclient () event function is registered, Sends a slave to all update operations in the process within the snapshot (slave buffer stored in the command). Again, when master transfers the Rdb file to slave, slave itself changes the state to redis_repl_connected, indicating that the copy has been completed and is in a state of real-time synchronization with master slave. The state transitions described above are shown in Figure 1-3, as the figure shows, standing in the slave role, when a network outage occurs, whether the slave itself is in redis_repl_connecting, Redis_repl_repl_transfer, or Redis_ repl_connected, the corresponding handler function is invoked to bring the slave into the Redis_repl_connect state, which means slave need to reboot the MASter sends the sync command to repeat the full volume synchronization process. The Redis_repl_wait_bgsave_start state in the figure is when the slave is connected to master (standing in Master's role), When master just had a snapshot process in the background and the RDB generated by the snapshot process was not suitable for the state that occurred when the slave was passed directly, the slave state was placed as Redis_repl_wait_bgsave_start. If there is a snapshot process at this time and another slave to initiate the snapshot process is found, simply copy the other slave's buffer content into the slave buffer and enter the Redis_repl_wait_bgsave_end state directly. If there is no background snapshot process at this time, slave goes directly into the Redis_repl_wait_bgsave_end state and initiates a background snapshot process. Figure 1-3 Redis-2.4.16 Master-slave replication state transition diagram The biggest problem in the above state transitions is that any network flash will cause slave to reconnect with master, and then re-enter the snapshot process, which will take a long time to retransmit the Rdb file, slave after receiving RDB files to restore RDB files to The memory is not serviced in the process (except the info command). Therefore, the provision of partial replication at least can be done in the network flash-break and update the command is not too many scenarios can be as far as possible to avoid a full copy of the scheme is particularly important.