Redis Deep Learning Notes (iii) RDB and AOF processes

An RDB is a way of redis persisted data, a Redis memory snapshot at a point in time, an RDB file loaded with Redis data restore, and a Redis master-slave data synchronization based on an RDB.

RDB Process:

1) Executing the bgsave command, the Redis parent process determines whether there is currently a child process being executed, such as the rdb/aof child process, if there is a bgsave command to return directly.

2) The parent process performs a fork operation to create a child process, the parent process is blocked during the fork operation, and the info Stats command is used to view the LATEST_FORK_USEC option to obtain the time-consuming, in microseconds, of the most recent fork operation.

3) After the parent process fork is finished, the Bgsave command returns the "Background saving started" message and no longer blocks the parent process, and can continue to respond to other commands.

4) The child process creates an RDB file, generates a temporary snapshot file based on the parent process memory, and then atomically replaces the original file after completion. Execute the lastsave command to get the last time the RDB was generated, corresponding to the Rdb_last_save_time option for the info statistic.

5) The process sends a signal to the parent process to indicate completion, and the parent process updates the statistics, as described in rdb_* related options under info persistence.

AoF is another way for Redis to persist data by logging each operation as a log, restoring all operations when Redis recovers data.

AOF process:

1) All write commands are appended to the AOF_BUF.

2) The AOF buffer is synchronized to the hard disk according to the corresponding policy.

3) as the aof file becomes larger, the aof file needs to be rewritten periodically for compression purposes.

4) When the Redis server restarts, you can load the aof file for data recovery.

AOF Buffer synchronization file policy (configuration Appendfsync parameter) Description:

Always: The command writes AOF_BUF after the call system Fsync operation synchronizes to the AoF file, Fsync the thread returns after completion

EVERYSEC: The system write operation is called after the command is written to Aof_buf, and the thread returns after write completes. Fsync synchronous file operations are called once per second by specialized threads

No: The command to write AOF_BUF after the call system write operation, do not fsync synchronization of aof files, synchronous hard disk operation by the operating system, usually the maximum synchronization cycle of 30s

When configured as always, each write is synchronized with the AoF file, and on a typical SATA hard drive, Redis can only support approximately hundreds of TPS writes, obviously contrary to the Redis high-performance features and is not recommended for configuration.

Configured as no, because the operating system each time the synchronization of the AoF file cycle is not controllable, and will increase the amount of data per synchronization of the hard disk, although improved performance, but data security is not guaranteed.

Configured as EVERYSEC, it is the recommended synchronization policy and is also the default configuration for both performance and data security. In theory, only 1 seconds of data are lost in the event of a sudden system outage.

(Policy selection is based on different business choices)

The system calls the write and fsync instructions:

The write operation triggers a deferred write (delayed write) mechanism, in which Linux provides page buffers to improve hard disk IO performance, write operations are returned directly after writing to the system buffers, and synchronous hard disk operations rely on system scheduling mechanisms such as:

The buffer page space is full or reaches a specific time period. Before synchronizing the files, if the system fails at this time, the data in the buffer will be lost.

Fsync for a single file operation (such as a aof file), to do a forced hard disk synchronization, Fsync will block until the write hard disk is completed and return to ensure data persistence.

The child process is generated by the fork operation, which takes up the same amount of memory as the parent process and theoretically requires twice times more memory to complete the persistence operation, but Linux has a write-time replication mechanism (copy-on-write). Parent-child processes share the same physical memory page, and when the parent process processes the write request, the page that is being modified is created as a copy, and the child process shares the entire parent process memory snapshot during the fork operation. Avoid making child process overrides on a large number of writes, which causes the parent process to maintain a large number of page replicas, resulting in memory consumption.

Linux kernel adds transparent Huge pages (THP) to the 2.6.38 kernel and supports page assignments for Huge page (2MB), which is turned on by default. When on, you can reduce the speed at which the fork creates the child process, but after the fork is executed, if THP is turned on, copying the page units from the original 4KB to 2MB will significantly increase the memory consumption of the parent process during the rewrite. It is recommended to set "sudo echonever>/sys/kernel/mm/transparent_hugepage/enabled" off THP.

Rdb and aof comparison:

RDB file Compact, RDB file generation and sub-process completion, does not block the main process, and can take advantage of multi-core CPU resources, data recovery is faster than AOF, but the Rdb method is easy to lose data, some companies in order to make full use of CPU resources, the REDIS process and CPU core binding, When an RDB is made, the child process and the parent process are resource-competitive and affect service throughput.

AOF more secure, can be more timely synchronization of data to the file, but aof need more disk IO expenditure, aof file size, file content recovery several relatively slow.

Note: The pictures in this article are from the book.

The next chapter describes the master-slave replication process

Redis Deep Learning Notes (iii) RDB and AOF processes