The persistence mechanism provided by Redis (RDB and AOF)

The persistence mechanism provided by Redis

Redis is a high-performance database that you can choose to persist or not persist.

If you want to save, there will be data synchronization problems, you can simply think of a piece of data in memory (snapshot), a piece of data on disk, Redis provides a very flexible persistence method:

Redis provides an RDB persistence and aof persistence, and the two mechanisms will be compared in this article

Advantages and slight application of the RDB mechanism

RDB persistence refers to writing a snapshot of an in-memory dataset to disk within a specified interval of time. is also the default persistence method, which is to write the in-memory data in a snapshot to the binary file, the default file name is Dump.rdb.

You can automatically make snapshot persistence by configuring settings. We can configure Redis to take snapshots automatically if more than M key is modified in n seconds, the following is the default snapshot save configuration

save 900 1 #If more than 1 key is modified within 900 seconds, initiate a snapshot save
    save 300 10 # 300 seconds If more than 10 keys are modified, a snapshot save is initiated
    save 60 10000 

RDB File Save Process

• Redis calls fork and now has child and parent processes.
• The parent process continues to process the client request, and the child process is responsible for writing the memory contents to the temporary file. Because the OS's write-time replication mechanism (copy on write) will share the same physical page, when the parent process processes the write request, the OS creates a copy of the page to be modified by the parent process, rather than writing the shared page. So the data in the child process's address space is a snapshot of the entire database at fork time.
• When the child process finishes writing the snapshot to the temporary file, replace the original snapshot file with a temporary file, and the child process exits.

The client can also use the Save or Bgsave command to notify Redis to do a snapshot persistence. The save action is to save the snapshot in the main thread, which blocks all client requests because Redis uses a main thread to process all client requests. So it is not recommended.

It is also important to note that each snapshot persistence is a full write of the memory data to disk once, not the incremental synchronization of only dirty data. If the amount of data is large, and more write operations, it will inevitably cause a large number of disk IO operations, may seriously affect performance.

Advantage

• Once this is done, your entire Redis database will contain only one file, making it easy to back up. For example, you may not want to archive some data for 1 days.
• Easy Backup, we can easily move an RDB file to other storage media
• An RDB recovers a large data set faster than AOF.
• An RDB maximizes the performance of Redis: The only thing the parent process has to do when saving an Rdb file is to fork out a child process, and the child process will handle all subsequent save work, and the parent process does not have to perform any disk I/O operations.

Disadvantage

• If you need to avoid losing data when the server fails, the RDB is not for you. Although Redis allows you to set different savepoint (save point) to control how often an RDB file is saved, it is not an easy operation because the Rdb file needs to preserve the state of the entire dataset. Therefore, you may be able to save the RDB file at least 5 minutes. In this case, you may lose several minutes of data in the event of a failure.
• Each time you save an RDB, Redis will fork () out a subprocess and perform the actual persistence work by the child process. When the dataset is large, fork () can be time consuming, causing the server to stop processing the client in such milliseconds, and if the data set is huge and CPU time is very tight, the stop time may even be a full second. Although the AOF rewrite also requires fork (), there is no loss of data durability regardless of the length of the AOF rewrite execution interval.

AOF File Save Process

Redis appends each received write command to the file via the Write function (default is appendonly.aof).

When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands saved in the file. Of course, because the OS caches write modifications in the kernel, it may not be written to disk immediately. The persistence of this aof method is also likely to lose some of the modifications. But we can tell Redis through the configuration file that we want to force the OS to write to disk through the Fsync function. There are three ways to do this (the default is: Fsync once per second)

appendonly yes // Enable aof persistence
# appendfsync always // Forcibly write to disk every time a write command is received, the slowest, but to ensure complete persistence, it is not recommended
appendfsync everysec // Forcibly write to disk once every second, a good compromise in performance and persistence is recommended
# appendfsync no // totally depends on os, the best performance, no guarantee of persistence

The AOF approach also poses another problem. Persistent files can become more and more large. For example, we call the INCR Test command 100 times, the file must save all 100 commands, in fact, 99 are redundant. Because you want to restore the state of the database, it is enough to save a set test 100 in the file.

In order to compress the aof persistence file. Redis provides the bgrewriteaof command. Receive this command Redis will use a snapshot-like method to save the in-memory data to a temporary file in the form of a command, and finally replace the original file. The specific process is as follows

• Redis Call Fork, now has parent-child two processes
• A child process writes a command to a temporary file to rebuild the state of a database based on a database snapshot in memory
• The parent process continues to process the client request, in addition to writing the write command to the original aof file. Cache the received write commands at the same time. This will ensure that if the child process rewrite fails, it will not be problematic.
• The child process signals the parent process when the child process writes the snapshot content to a temporary file that has been written to the command mode. The parent process then writes the cached write command to the temporary file as well.
• Now the parent process can replace the old aof file with the temporary file and rename it, and the subsequent write commands are appended to the new AoF file.

Note that the operation to rewrite the aof file does not read the old aof file, but instead overwrites the entire in-memory database content with a new aof file in the form of a command, which is a bit similar to a snapshot.

Advantage

• Using AOF persistence makes Redis very durable (much more durable): You can set different fsync policies, such as no fsync, one fsync per second, or Fsync each time you execute a write command. The default policy for AOF is Fsync once per second, in which Redis can still maintain good performance and, even if there is a failure, it will lose at most one second of data (Fsync will execute in a background thread, so the main thread can continue to work on the command request).

• The AOF file is an append-only log file (append only), so the write to the AOF file does not require seek, even if the log contains incomplete commands for some reason (such as a write-in disk full, write-down, etc.), The Redis-check-aof tool can also easily fix this problem.
  Redis can automatically override AOF in the background when the AOF file size becomes too large: The rewritten new AOF file contains the minimum set of commands required to restore the current dataset. The entire rewrite operation is absolutely secure, as Redis will continue to append commands to the existing AOF file during the creation of the new AOF file, and the existing AOF files will not be lost even if there is an outage during the rewrite. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and begins appending to the new AOF file.

• The AOF file preserves all write operations performed on the database in an orderly manner, and these writes are saved in the format of the Redis protocol, so the contents of the AOF file are easily readable and parsing (parse) is easy. The export AOF file is also very simple: for example, if you accidentally executed the Flushall command, but as long as the AOF file is not rewritten, just stop the server, remove the Flushall command at the end of the AOF file, and restart Redis, you can The data set reverts to the state before Flushall execution.

Disadvantage

• For the same dataset, the volume of the AOF file is usually larger than the size of the RDB file.

• Depending on the Fsync policy you are using, the AOF may be slower than the RDB. In general, the performance of Fsync per second is still very high, while closing fsync can make AOF as fast as an RDB, even under high load. However, the RDB can provide a more guaranteed maximum delay time (latency) when handling large write loads.

• AOF has been a bug in the past: Because of the reason of individual commands, the AOF file cannot be restored as it was when the data set was reloaded. (for example, a blocking command Brpoplpush has caused such a bug.) The test suite adds tests to this situation: they automatically generate random, complex datasets, and re-load the data to make sure everything is fine. Although this bug is not common in AOF files, it is almost impossible for an RDB to appear in comparison.

Choice

In general, if you want to achieve data security that is comparable to PostgreSQL, you should use two persistence features at the same time.

If you are very concerned about your data, but can still tolerate data loss within a few minutes, you can use only the RDB persistence.

The rest of the situation I personally prefer to choose AOF

5.1.

1). RDB Persistence

The mechanism is to write a snapshot of the in-memory data set to disk within a specified interval of time. For example, every 15 minutes there are data changes that synchronize the data in memory with the disk.

This method is used in the Redis default configuration, as follows:

# After the SEC (min) If at least 1 key changed 15 minutes if more than 1 content has changed, perform the Save

# After the SEC (5 min) If at least changed 5 minutes if more than 10 content has changed, perform the Save

# after the SEC if at least 10000 keys changed 1 minutes if more than 10,000 of the content has changed, perform the Save

2). AoF Persistence:
This mechanism will record every write processed by the server as a log, and read the file in the early of the Redis server to rebuild the database to ensure that the data in the database is complete after startup.
3). No persistence:
We can disable the persistence of the Redis server by configuring it so that we can treat Redis as a feature-enhanced version of memcached.
4). Apply both AoF and RDB.

5.2. Advantages and disadvantages of the RDB mechanism:

Advantages
1). Once this is used, your entire Redis database will contain only one file, which is perfect for file backups.

For example, you might want to archive the last 24 hours of data every hour and archive the last 30 days of data every day.

With such a backup strategy, we can recover very easily once the system has a catastrophic failure.
2). The RDB is a great choice for disaster recovery.

Because we can easily compress a single file and then transfer it to other storage media.
3). Maximum performance.

For a Redis service process, the only thing it needs to do when it starts to persist is to fork out the subprocess, and then the child processes to do the persistent work.

This can greatly prevent the service process from performing IO operations.
4). If the data set is large, the RDB will be more efficient to start than the AOF mechanism.

Disadvantages
1). An RDB is not a good choice if you want to ensure high availability of data, which minimizes data loss.

Since the system is down before the scheduled persistence, the data that has not been written to the disk before is lost.
2). Since the RDB assists with data persistence through the fork process, it can cause the entire server to stop serving hundreds of milliseconds, or even 1 seconds, if the data set is large.

5.3. Advantages and disadvantages of AOF mechanism:

Advantages
1). This mechanism can result in higher data security, i.e. data persistence.

There are 3 synchronization policies available in Redis, that is, synchronization per second, synchronization per modification, and unsynchronized.

In fact, synchronization per second is also done asynchronously, and its efficiency is very high, the difference is that once the system is down, then the data will be lost in a second.

Each time we modify the synchronization, we can treat it as a synchronous persistence, that is, each occurrence of the data changes will be immediately logged to disk.

It can be predicted that this is the least efficient way.

As for the no-sync, needless to say, I think we can understand it correctly.
2). Because this mechanism writes to log files in append mode, it does not break the content that already exists in the log file even if there is an outage during the write process.

However, if we only write half of the data this time, there is a system crash problem, do not worry,

Before the next boot of Redis, we can use the Redis-check-aof tool to help us solve the problem of data consistency.
3). If the log is too large, Redis can automatically enable the rewrite mechanism. That is, Redis continuously writes the modified data to the old disk file in Append mode.

Redis also creates a new file to record which modification commands are executed during this period. Therefore, the data security can be better ensured when the rewrite is switched.
4). AOF contains a well-formed, easy-to-understand log file for recording all modification operations. In fact, we can also complete the reconstruction of the data through this file.

What are the disadvantages of aof?
1). aof files are typically larger than the Rdb file for the same number of datasets.
2). Depending on the synchronization strategy, the AOF is often slower than the RDB in terms of operational efficiency. In summary, the efficiency of the synchronization policy per second is high, and the efficiency of the synchronization disable policy is as efficient as the RDB.

5.4. Other

5.4.1. Snapshotting:
By default, Redis dumps a snapshot of the dataset to the Dump.rdb file. In addition, we can modify the frequency of the Redis server dump snapshot through the configuration file, after we open the 6379.conf file, we search for save, we can see the following configuration information:
Save 1 #在900秒 (15 minutes), dump memory Snapshot if at least 1 key changes occur.
After save #在300秒 (5 minutes), dump memory Snapshot if at least 10 key changes occur.
After save 10000 #在60秒 (1 minutes), dump memory Snapshot if at least 10,000 key changes occur.

5.4.2. Dump snapshot mechanism:
1). Redis first fork the child process.
2). The child process writes the snapshot data to the temporary RDB file.
3). After the child process completes the data write operation, replace the old file with the temporary file.

5.4.3. aof file:
It has been said many times that the RDB snapshot timing dump mechanism does not guarantee good data persistence. If our application is really concerned about this point, we can consider using the AOF mechanism in Redis. The default mechanism for Redis servers is the RDB, and if you need to use AOF, you need to modify the following entries in the configuration file:
Change AppendOnly No to appendonly Yes
From now on, Redis appends the data modification commands to the AoF file every time it receives a command. The next time Redis restarts, the information in the AoF file needs to be loaded to build the most up-to-date data into memory.

5.4.5. AOF configuration:
There are three synchronization modes in the Redis configuration file, namely:
Appendfsync always #每次有数据修改发生时都会写入AOF文件.
Appendfsync everysec #每秒钟同步一次, this policy is the default policy for AOF.
Appendfsync no #从不同步. Efficient but data is not persisted.

5.4.6. How to fix the AoF file for bad damage:
1). Make an additional copy of the existing AoF file that has been damaged.
2). Execute the "redis-check-aof--fix <filename>" command to repair the damaged AoF file.
3). Restart the Redis server with the repaired aof file.

5.4.7. Redis Data Backup:
In Redis, we can back up the running Redis data files online by copy. This is because the Rdb file will no longer be modified once it is generated. Each time Redis dumps the latest data into a temporary file, it then renames the temporary file to its original data file name using the Rename function atomicity. So we can say that copy data files are safe and consistent at any time. For this reason, we can periodically back up Redis data files by creating cron jobs and copy the backup files to secure disk media.

5.5. Write now

// Save immediately, save synchronously
     public static void syncSave () throws Exception {
         Jedis jedis = new Jedis ("127.0.0.1", 6379);
         for (int i = 0; i <1000; i ++) {
             jedis.set ("key" + i, "Hello" + i);
             System.out.println ("Set data of key" + i + "to redis");
             Thread.sleep (2);
         }
         // Execute the save, a dump.rdb database file will be generated under the server
         jedis.save ();
         jedis.close ();
         System.out.println ("write completed");
     }

Operation Result:

The Save method here is synchronous, and no subsequent code is executed until the write is complete.

5.6. Asynchronous Write

// Asynchronous save
     public static void asyncSave () throws Exception {
         Jedis jedis = new Jedis ("127.0.0.1", 6379);
         for (int i = 0; i <1000; i ++) {
             jedis.set ("key" + i, "Hello" + i);
             System.out.println ("Set data of key" + i + "to redis");
             Thread.sleep (2);
         }
         // Perform asynchronous save, a dump.rdb database file will be generated under the server
         jedis.bgsave ();
         jedis.close ();
         System.out.println ("write completed");
     } 

If the amount of data is very large, to save a lot of content, we recommend the use of Bgsave, if the content is small, you can use the Save method. The comparison of the various methods originates from the Netizen's blog.

The persistence mechanism provided by Redis (RDB and AOF)