Redis Research (11)-Data persistence

first, the persistence of
the strength of Redis is largely due to the fact that it stores all of the data in memory, and in order for Redis to ensure that data is not lost after a reboot, the data needs to be synchronized from memory to the hard disk in some form, which is a process of persistence.

Redis supports two ways of persistence, one for RDB and one for aof. You can use either one alone or combine the two.

1. Rdb Mode

The RDB approach is persisted through snapshots (snapshotting), and Redis automatically snaps all of the data in memory and stores it on the hard disk when certain conditions are met. The conditions for a snapshot can be customized by the user in the configuration file, consisting of two parameters: the number of times and changes in the key. A snapshot is taken when the number of keys changed within a specified amount of time is greater than the specified number. The RDB is the default persistence for Redis, with 3 conditions already provisioned in the configuration file:

Save  1save  10save  60 10000

The save parameter specifies the snapshot condition, which can have more than one condition, and the relationship between the conditions is "or". As stated above, save 900 1 means that at least one key is changed in 15 minutes (900 seconds) to take a snapshot.

If you want to disable automatic snapshots, simply delete all of the save parameters.

Redis stores the snapshot files in the current directory's Dump.rdb file by default, and you can specify the storage path and file name of the snapshot files by configuring Dir and dbfilename two parameters respectively.

clearing the Redis implementation snapshot process is a great help in understanding the features of snapshot files. The process for snapshots is as follows.
(1) Redis uses the fork function to copy a copy of the current process (the parent process) (child process);
(2) The parent process continues to receive and process commands from the client, and the child process begins to write the data in memory to a temporary file on the hard disk;
(3) When the child process has finished writing all the data, it will replace the old Rdb file with the temporary file, at which time the snapshot operation is completedIn the form.

The operating System (Unix-like operating system) uses the write-time replication (Copy-on-write) policy when the fork is executed, that is, when the fork function occurs, the parent-child process shares the same memory data, and when the parent process changes one of the pieces of data (such as executing a write command), The operating system copies the piece of data to ensure that the child process's data is not affected, so the new Rdb file stores the memory data that executes the fork moment.

The above procedure shows that Redis does not modify the Rdb file during a snapshot, only the old file is replaced with the new one at the end of the snapshot, which means that the Rdb file is complete at any time. This allows us to backup the Redis database by regularly backing up the RDB files. An RDB file is a binary format that is compressed (you can configure the Rdbcompression parameter to disable compression to conserve CPU), so the space used is less than the size of the in-memory data and is more conducive to transmission.

In addition to automatic snapshots, you can manually send the Save or bgsave command to have Redis perform a snapshot, the difference between the two commands is that it is a snapshot operation by the main process, which blocks other requests, which take a snapshot operation through the fork subprocess.

After Redis boots, the Rdb snapshot file is read and the data is loaded into memory from the hard disk. This time is different depending on the size of the data and the structure and server performance.

It typically takes 20-30 seconds to load a snapshot file with 10 million string type keys and a size of 1GB into memory.

By using the Rdb method to persist, all data changed after the last snapshot is lost once the Redis exits unexpectedly. This requires the developer to control the possible loss of data in an acceptable range by combining the set of automatic snapshot conditions in a specific application. If the data is important enough to withstand any loss, consider using the AoF method to persist.

2. AOF mode

By default, Redis does not have persistent aof (append only file) mode, which can be turned on by appendonly parameters:

AppendOnly Yes

When AOF is turned on for each execution of a command that changes data in Redis, Redis writes the command to the AoF file on the hard disk. The aof file is saved in the same location as the Rdb file, which is set by the Dir parameter, and the default file name is Appendonly.aof, which can be modified by the Appendfilename parameter:

Append file name  appendonly.aof

The following is a detailed implementation of AOF persistence, assuming that the following 4 commands are executed with the AOF persistence enabled:

SET foo 1SET foo 2SET foo 3GET foo

Redis writes the first 3 commands to the AoF file, at which point the contents of the AoF file are as follows:

*2$6<strong>select</strong>$1<strong>0</strong>*3$3<strong>set</strong>$3 <strong>foo</strong>$1<strong>1</strong>*3$3<strong>set</strong>$3< Strong>foo</strong>$1<strong>2</strong>*3$3<strong>set</strong>$3<strong >foo</strong>$1<strong>3</strong>

The visible aof file is a plain text file whose content is the content of the original communication protocol that the Redis client sends to Redis (for readability,This shows the actual command part in bold), it is visible that Redi s does only record the first 3 commands. However, one problem is that the first 2 commands are actually redundant because the results of the two execution are overwritten by the third command. As more commands are executed, the size of the aof file becomes larger, even though the actual data in memory may not be much. Naturally, we want redi s to be able to automatically optimize the AoF file, as in the above example, delete the first two useless records, leaving only the third one. That's exactly what Redis does, and Redis automatically rewrites the aof file whenever certain conditions are met, which can be set in the configuration file:

Auto-aof-rewrite-percentage 100auto-aof-rewrite-min-size  64MB

The meaning of the Auto-aof-rewrite-percentage parameter is that when the current aof file size exceeds the size of the AoF file at the time of the last rewrite, it is rewritten again, based on the aof file size at startup if it was not previously rewritten.

The auto-aof-rewrite-min-size parameter limits the minimum aof file size that is allowed to be overridden, usually even though there are a lot of redundant commands in aof files that we don't care much about.

In addition to having Redis automate overrides, we can also proactively perform aof overrides manually using the Bgrewri teaof command.

The following rewrite of the aof file in the example above is:

*2$6<strong>select</strong>$1<strong>0</strong>*3$3<strong>set</strong>$3 <strong>foo</strong>$1<strong>3</strong>

The Visible redundancy command has been removed. The process of rewriting is only relevant to the data in memory, not to the previous aof file, which is similar to the RDB, except that the file format is completely different.

at startup, Redis executes the commands in the AoF file one by one to load the data from the hard disk into memory, which is slower than the Rdb .

It is important to note that although AOF will record commands in the aof file every time the operation is made to change the contents of the database, in fact, because of the caching mechanism of the operating system, the data is not actually written to the hard disk, but rather into the system's hard disk cache. By default, the system performs a synchronous operation every 30 seconds, so that the contents of the hard disk cache are actually written to the hard disk, which can cause data loss in the hard disk cache if the system exits unexpectedly during this 30-second process. In general, applications that enable AOF persistence cannot tolerate such losses, requiring redis to proactively require the system to synchronize cached content to the hard disk after writing to the AoF file.

In Redis we can set the timing of synchronization with the Appendfsync parameter:

# Appendfsync Alwaysappendfsync everysec# Appendfsync No

By default, Redis takes the everysec rule, which is to perform a synchronous operation once per second. Always means that synchronization is performed every time a write is performed, which is the safest and slowest way. No means that the synchronous operation is not active, but rather is done entirely by the operating system (i.e. every 30 seconds), which is the fastest but least secure way. In general, using the default value of Everysec is sufficient, both for performance and for security.

Redis allows both AOF and RDB to be enabled, ensuring data Security (AOF) and making backup (RDB) operations easy. After you restart Redis, Redis uses the aof file to recover the data, because the aof way of persisting may lose less data.

Redis Research (11)-Data persistence