Redis (14) The principle of master-slave replication of Redis

One: the principle and steps of redis master-slave replication.

 
Step 1: Copy initialization
---> After starting from redis, it will send a SYNC command to the main redis according to the configuration. After version 2.8, the PSYNC command is sent.
---> After the master redis receives the SYNC command, it starts to save the snapshot file in the background (that is, the RDB persistence process), and caches the commands received during saving the snapshot.
---> After the master redis completes the snapshot, the master redis sends the snapshot file and the cache command to the slave redis. Copy initialization is complete.
---> When the replication initialization of the master redis is completed, the master redis will send the write command asynchronously to the slave redis whenever it receives the write command.
 
 
Step 2: The synchronization process
---> After sending SYNC from redis, after receiving the snapshot file (RDB) and cache command, write the content to a temporary file on the hard disk
---> After writing the snapshot file and cache command from redis, it will replace the RDB file with a temporary file according to the slave configuration file (determined by the two parameters dir and dbfilename). And execute the command. So as to achieve the same master-slave redis content.
---> The replication synchronization phase runs through the entire master-slave synchronization process until the master-slave relationship ends.
 
 
The following figure simulates the master-slave replication process.

 
 
Two: An important issue in the principle of redis master-slave replication.
The first: Is the data synchronization of the master and slave redis asynchronous or synchronous?
---> Redis master-slave replication, called optimistic replication. Is asynchronous. The content of the master-slave database is tolerated for a certain period of time. But the data of the two will eventually be synchronized.
---> Since it is asynchronous, there will be a time window where the master and slave redis data are inconsistent.
 
 
Second: When the master redis receives the write command, before sending a synchronous command to the slave redis, the master and slave redis network is disconnected, how to solve it.
---> redis provides a restriction strategy. To prevent the master-slave from disconnecting, the master redis cannot know whether its write command is completely sent to the slave redis
---> Two configuration parameters: min-slaves-to-write and min-slaves-max-lag
---> The number n after the min-slaves-to-write parameter is to ensure that the master redis will provide external write services when at least n or more slave redis are connected to the master redis normally.
---> min-slaves-max-lag The number m (unit: second) after the parameter, the longest lost time between the master and slave redis. The difference between the time when the slave redis last contacted the master redis (that is, sending the REPLCONF ACK command) to the current time is less than this m. The master redis believes that the master-slave connection is normal and provides write services to the outside. Once it is greater than this value, it indicates that there is a disconnection from redis without providing write operations to the outside world.
---> This feature ensures that the master and slave data are inconsistent under the network partition. This feature is off by default.
 
The third: master-slave disconnected, how does the master-slave achieve data synchronization after reconnecting.
---> Before version 2.6, every time you re-practice, you need to complete a complete synchronization of the snapshot file and the cache command, and start again from the beginning. When the amount of data is large, the synchronization efficiency will be low.
---> Redis2.8 new version, new strategy. Able to support [conditional] incremental data transmission, it is only necessary to transmit the commands without synchronization during the out-of-connection period.
 
Fourth: Can the master-slave database realize the business scenario of read-write separation?
---> can be achieved
 
 
Fifth: Hang down from redis, restart, master redis will synchronize data to slave redis. Then the master redis hangs up, how to achieve data recovery and the cluster continue to provide external write services?
---> Two schemes: (1) manual recovery (2) sentry strategy
---> Manual recovery
1. Use the SLAVEOF NO ONE command in the slave data to upgrade from the redis database to the master database to continue providing services.
2. Start the master redis that crashed before, and then use the SLAVEOF command to set it as the slave database of the new master redis, so that the data can be synchronized back.
3. Note: When replication is enabled and the main redis database is disabled, do not use Supervisor and similar process management tools to automatically restart the main database after it crashes, and avoid upgrading one of the redis to the main redis. Manually restart the old master redis before, because after the old redis is started, because there is no persistence and the data in the database is empty, all slave redis nodes will also be synchronized to empty.
 
 
Three: the distributed characteristics of redis
---> Separation of read and write to achieve the scenario of reading more and writing less
 
---> The primary database is not persistent, and the secondary database is persistent.
(1) However, there will be a hidden danger that after the master is restarted, there will be no data recording due to non-persistence, and the slave data will also be emptied. This master-slave structure requires upgrading one slave first, and then starting the old master database.
 
---> No hard disk copy.
(1) The purpose is to reduce hard disk reads and writes and improve performance.
(2) When the master redis copies the snapshot file (generating the snapshot RDB), it is not written to the hard disk, but transferred directly to the slave redis database through memory. After the current 2.8 version, this mode can be enabled through a configuration file, which belongs to the test experiment stage. The repl-diskless-sync yes option is enabled.
 
---> Incremental replication.
(1) Objective: After the master-slave redis is disconnected, the master redis does not need to implement a snapshot of all data and command cache each time to synchronize with the slave redis on the re-connection of the lost contact. Instead, it only synchronizes data changes during the out-of-connection period, which improves performance.
(2) The basis for achieving incremental replication
==> The slave redis stores the run id of the master redis. Each redis instance will have a unique run ID. When the instance is restarted, a new run ID will be automatically generated
==> In the replication synchronization phase, each time the master redis sends a command to the slave redis, it will store the command in a backlog at the same time, and record the offset range of the current backlog queue to store the command.
==> When redis receives the command from the main redis, it will also record the offset of the command.
(3) The process of achieving incremental replication
==> When the slave database needs the master database for replication, after version 2.8, the slave redis will send [PSYNC + master redis run id + latest command offset before disconnection] to the master redis
==> After receiving the command, the master redis determines whether the master redis running id and the running id are the same. At the same time, it is determined whether the latest command offset before the disconnection is in its own backlog queue. If it exists, then from its own backlog queue, it decides how many commands to copy to the slave redis, thereby achieving incremental replication.
==> If the running id of the main redis sent from redis is different from its own running id, or the running id is the same, but the offset is not in the backlog queue of the main redis. Full copy is performed. (Generate snapshot file + cache command)
==> backlog queue size configuration: repl-backlog-size specified (default 1mb),
==> Release time of backlog queue: specified by repl-backlog-ttl. That is, when there is a disconnection between the redis and the main redis, how long does it take to release the memory space of the backlog queue. The default is 1 hour.