Redis replication principle and Analysis

1. Test

See the master-slave test post.

2 Principle First,The implementation of slave synchronization to the master is:

Slave sends a synchronization request to the master (Send sync command), The Master first dump out the RDB file, and then transfers all the RDB files to the slave, then the master forwards the cached write commands to the slave, the first synchronization is completed.
Second,And the future synchronization implementation is:
The master directly sends the snapshots of the variables to each slave in real time.
However, no matter what causes the slave and master to be disconnected and reconnected, the above two steps will be repeated.
Redis master-slave replication is based on the persistence of memory snapshots. As long as there is a slave, there will be a memory snapshot.

)

However, we can clearly see that RDB has its shortcomings, that is, once redis encounters a problem, the data stored in our RDB file is not brand new, data from the last RDB file generation to redis downtime is all lost. In some businesses, this is tolerable. We also recommend that these businesses use RDB for persistence, because the cost of enabling RDB is not high. However, for other applications that require extremely high data security and cannot tolerate data loss, RDB is powerless. Therefore, redis introduces another important persistence mechanism: aof log, which will be analyzed later.

3. RDB snapshot principle redis supports the persistence mechanism of saving the snapshot of the current data into a data file, that is, the RDB snapshot. This method is quite understandable, but how does one generate snapshots for a continuously written database?

Redis uses the copy on write mechanism of the fork command (private memory is not shared memory ). When a snapshot is generated, the current process is fork out of a sub-process, and then all the data is recycled in the sub-process to write the data into an RDB file.

We can configure the RDB snapshot generation time using the Save command of redis. For example, you can configure to generate a snapshot when there are 100 writes within 10 minutes, you can also configure to generate a snapshot when there are 1000 writes within one hour, or you can implement multiple rules together. These rules are defined in the redis configuration file. You can also use the redis config set command to set rules during redis running without restarting redis.

Configure in redis:

1. save 900 1 # RDB is generated when a keys data record is changed within 900 seconds;
2. save 300 10 # RDB is generated when 10 keys data entries are changed within 300 seconds;
3. save 60 10000 # RDB is generated when 10000 keys data entries are changed within 60 seconds;

################################ SNAPSHOTTING  ################################### Save the DB on disk:##   save <seconds> <changes>##   Will save the DB if both the given number of seconds and the given#   number of write operations against the DB occurred.##   In the example below the behaviour will be to save:#   after 900 sec (15 min) if at least 1 key changed#   after 300 sec (5 min) if at least 10 keys changed#   after 60 sec if at least 10000 keys changed##   Note: you can disable saving at all commenting all the "save" lines.save 900 1save 300 10save 60 10000

 

4. redis aof log

The full name of aof logs is append only file. We can see from the name,It is an append log file. Unlike the BINLOG of a general database, the aof file is identifiable plain text.It contains standard redis commands. Of course, not all commands sent to redis must be recorded in the aof log. Only commands that may cause data changes will be appended to the aof file.

 

So every command for data modification generates a log. Will the aof file be large?
The answer is yes, the aof file will become larger and larger, so redis provides another function called aof rewrite (you can use redis to provide the bgrewriteaof command ). Its function is to regenerate an aof file. The operation of a record in the new aof file will only be performed once, unlike an old file, operations on the same value may be recorded multiple times. The generation process is similar to that of RDB. It is also a fork process that directly traverses data and writes data to a new aof temporary file (this process is similar to that of RDB, however, the data is split into a write command ). During the process of writing new files, all write operation logs will still be written to the old aof file and recorded in the memory buffer. After the operation is completed, all logs in the buffer zone are written to the temporary file at one time. Then, call the atomic RENAME command to replace the old aof file with the new aof file. (In this way, the old aof can restore the memory. If a new aof is generated, the old one will not exist. The new aof file can be used to restore the memory, this solves the increasing aof problem .) Aof is a file write operation. Its purpose is to write operation logs to the disk. Therefore, it will also encounter five write operations.

 

How high is the security of aof writing?

In fact, this can be set. After calling write (2) for aof in redis, when will fsync be called to write it to the disk and controlled through the appendfsync option, the following three appendfsync settings gradually increase the security intensity.

1) appendfsync No
When appendfsync is set to no, redis does not actively call fsync to synchronize the aof log content to the disk, so all of this depends on the operating system debugging. For most Linux operating systems, fsync is performed every 30 seconds to write data in the buffer zone to the disk.

2) appendfsync everysec
When appendfsync is set to everysec, redis performs an fsync call every second by default to write data in the buffer zone to the disk. However, when the fsync call lasts for more than 1 second. Redis will adopt the fsync delay policy and wait a second. That is to say, if you perform fsync two seconds later, this fsync will be executed no matter how long it will take. At this time, because the file descriptor will be blocked during fsync, the current write operation will be blocked.

The conclusion is that in most cases, redis performs fsync every second. In the worst case, an fsync operation is performed in two seconds. This operation is called group commit in most database systems. It combines the data of multiple write operations and writes logs to the disk at one time.

3) appendfsync always
When appendfsync is set to always, fsync is called for every write operation, which is the safest data. Of course, because fsync is executed every time,
Therefore, its performance will also be affected.

Redis data recovery:
The RDB startup time is shorter for two reasons:
1. Each piece of data in the RDB file has only one record, and there may be no multiple operation records of the same data as the aof log. Therefore, you only need to write each data entry once.
2. The storage format of RDB files is the same as the encoding format of redis data in the memory. data encoding is not required. Therefore, the CPU consumption is much less than that of aof logs.

5. redis RDB File

Run the sync command on the slave server to request synchronization. The following code returns the RDB file, which is a binary file,

redis 127.0.0.1:6380> sync"REDIS0002\xfe\x00\n\anumbers\x0f\x0f\x00\x00\x00\n\x00\x00\x00\x01\x00\x00\xc0x03\x00\xff\x00\x03old\xc0\x01\x00\bkeywatch\xc0\x03\x00\x04name\x05kerry\x00\x03aaa\xc0o\x02\aletters\x02\x01c\x01b\t\x04car1\x19\x02\x04name\x05\x00AUDIO\x05price\x03\x0030w\xff\t\x04car2\x19\x02\x04name\x05\x00AUDIO\x05price\x03\x0020w\xff\t\x04car3\x19\x02\x04name\x05\x00buick\x05price\x03\x0010w\xff\x00\x03key\x03aaa\x00\x03num\xc0\x04\x00\x02a1\xc0\x01\x00\x02a2\xc0\x02\x00\x06keynew\xc0\x04\x00\x02a3\xc0\x03\x02\bletters2\x03\x01c\x01d\x01e\x00\acompany\x03alu\x00\x0coldvalue=GET\x03old\xff"redis 127.0.0.1:6380>

The specific file is under the redis directory, as shown below:

6. redis aof File

Disabled by default. Enable the aof setting as follows:

############################## APPEND ONLY MODE ################################ By default Redis asynchronously dumps the dataset on disk. If you can live# with the idea that the latest records will be lost if something like a crash# happens this is the preferred way to run Redis. If instead you care a lot# about your data and don‘t want to that a single record can get lost you should# enable the append only mode: when this mode is enabled Redis will append# every write operation received in the file appendonly.aof. This file will# be read on startup in order to rebuild the full dataset in memory.## Note that you can have both the async dumps and the append only file if you# like (you have to comment the "save" statements above to disable the dumps).# Still if append only mode is enabled Redis will load the data from the# log file at startup ignoring the dump.rdb file.## IMPORTANT: Check the BGREWRITEAOF to check how to rewrite the append# log file in background when it gets too big.appendonly yes

It cannot be generated in windwos, and there may be compatibility problems. The generated aof file is in the same directory as the RDB file.

Redis replication principle and Analysis