Redis Persistence Grooming

1 Introduction

Redis supports both RDB and aof two persistence mechanisms, which persist to avoid data loss due to process exit, and can be implemented by using previously persisted files on reboot.
Recovery of the data. 2 RDB

RDB persistence saves the current process data generation snapshot to the hard disk, represents a snapshot of the data for Redis at a point in time, and the RDB has both manual and automatic triggering.
-Manual Trigger
-Save: Blocks the server until the RDB is complete and is deprecated;
-The Bgsave:redis process fork out the subprocess, the RDB persistence process is responsible by the child process and ends automatically when completed. The RDB internal to Redis uses Bgsave.
-Automatic triggering
Internal automatic triggering RDB persistence has: ①save m n configuration (n changes in M seconds, automatically trigger Bgsave), ② from the node to perform full-volume replication, the master node automatically executes Bgsave, generate files sent to the slave node; ③debug reload automatically triggers ' save '; ④ default shutdown, if AOF is not turned on, automatic execution bgsave. 2.1 Bgsave is the main triggering RDB persistence mode

The process is as follows:
Bgsave-> Parent Process->fork-> child process--generates an RDB file--The parent process executes the Bgsave,redis parent process to determine if there is a child process being executed (eg. Rdb/aof child process), the presence is directly returned; the parent process Fork creates the child process, and the parent process blocks during the fork (Info stats View Latest_fork_usec, the most recent fork operation takes time-microseconds); When the fork is complete, Bgsave returns " Background saving started ", no longer blocks the parent process and can continue to respond to other commands; The child process creates an RDB file, generates a temporary snapshot file based on the parent process memory, and then atomically replaces the original file (Lastsave rdb_ Last_save_time, the time the RDB was last generated); The signal notifies the parent process to update the statistics.

The Dir:rdb file of the configuration file holds the directory, Dbfilename: its file name.

When the bad disk or disk is full, modify config set Dir{newdir} online to the new available disk path, and then bgsave the disk switchover (RDB, aof apply).
Similarly, the online config set Dbfilename{newfilename} is available.
Redis uses the LZF algorithm to compress RDB files, which is much smaller than memory data, consumes CPU but significantly ↓ file volume, facilitates saving or network transmission, on-line recommendation on (default on). (config set rdbcompression{yes|no}) 2.2 Advantages and disadvantages

Excellent: ① Compact Compressed binary file, can perform bgsave backup every x hours, disaster recovery; ② loading RDB recovering data is much faster than aof.
missing: ① cannot be persisted in real time (Bgsave each time to fork the sub-process, heavyweight operation); ② in a specific binary format, the old and new versions are incompatible. 3 AOF

AOF (append only file) persists, the log records each write command, restarts, and then re-executes the commands in the AOF file, recovering the data. It solves the real-time nature of data persistence. Understanding and mastering its mechanisms, ↑ a balance between data security and performance.

Turn on Aof:appendonly Yes (default does not open)
AOF file name: Appendfilename (default file name appendonly.aof)
Save path: Dir (same RDB) 3.1 aof is the mainstream way of Redis persistence

AOF Workflow:
Command Write (Append)->aof buffer--File sync (sync)->aof file--file rewrite (rewrite) <-restart load (load)

All write commands are appended to the AOF_BUF (buffer); AoF buffers are synchronized to the hard disk according to corresponding policies; As the aof file becomes bigger and larger, the aof file needs to be rewritten periodically to achieve the compression purpose; When the Redis server restarts, the AoF file can be loaded for data recovery.
Command Write
AOF directly using Text protocol format

① text protocol compatibility is very good; after ② on aof, all write commands include append operation, directly adopt protocol format, avoid two processing overhead, ③ readability, convenient for direct modification and processing.
AoF why the command was appended to the AOF_BUF (buffer).
①redis Single-threaded response command, directly appended to the hard disk, performance depends entirely on the current hard disk load, ② can provide a variety of buffer synchronous drive strategy, performance and security balance.
-File synchronization
Redis provides a variety of aof buffer synchronization file policies, controlled by parameter appendfsync.
Always: No configuration is recommended. Each write must be synchronized, the cost is high;
No: Performance ↑, but data security cannot be guaranteed;
EVERYSEC: Recommended configuration, default configuration. Balance performance and data security. Theoretically a sudden outage of the system loses 1 seconds of data.
-Override mechanism
As commands continue to be written, the aof file becomes larger, and the rewrite mechanism compresses the file volume. Rewrite the in-process data-to-write command, Sync-and-new aof file.

The volume becomes smaller after rewriting and can be loaded faster by Redis

① has timed out not to write, ② the new aof file retains only the final data of the Write command, ③ Multiple write commands are merged into one (Eg.lpush list A, lpush list B...->lpush List A b ...), anti-client buffer overflow.

Manual trigger: Bgrewriteaof
Auto Trigger: Auto-aof-rewrite-min-size, Auto-aof-rewrite-percentage

Bgrewriteaof-> Parent Process->fork-> (aof_buf-> old aof file,aof_rewrite_buf-> new aof file, child process, signal notification parent process, new AoF file

1. Execute the AOF rewrite request, if the bgsave is being executed, the rewrite command is deferred until Bgsave is complete;
2. Parent process fork child process;
3.1 Fork, continue to respond, write aof buffer +appendfsync policy to synchronize the hard disk, ensure correctness, 3.2 fork (write-time replication technology), the child process can only share the fork operation of the memory data. Save new data with aof rewrite buffer;
4. The child process writes the new AoF file according to the command consolidation rule based on the memory snapshot;
5.1 After writing to the new file, notify the parent process;
5.2 The parent process writes the data of the AOF rewrite buffer to the new aof file;
5.3 Replace the old file, complete the rewrite.
-Reboot Load
Persistent file loading process:

1.AOF persistent open and the presence of AOF file, the priority load AoF file, 2.AOF shutdown or aof file does not exist, loading the Rdb file; 3. After the load succeeds, Redis starts the print failure log.
-File Check
Startup is refused when loading a corrupted aof file. At this point, first back up, then redis-check-aof--fix repair, and finally diff-u contrast differences, manual modification of the completion (incomplete). Aof-load-truncated (default on) ignores and continues to start Redis when the power is suddenly dropped and the aof tail is not complete. 3.2 Problem Location optimization

Fork

Fork copies the spatial memory page table of the parent process, EG.10GB Redis, which requires a complex control of the 20MB memory page table, and the fork time is proportional to the total amount of progress in the process.
Improvement:
① priority physical machine; ② control the maximum available memory for Redis instances (within 10G of Single instance), ③ reasonable configuration of Linux memory allocation strategy, ④ reduce fork frequency, ↓ full copy. Child process Overhead
The child process is responsible for aof| The rewriting of RDB files mainly involves CPU, memory, hard disk consumption.
CPU:
Multiple Redis instances to ensure that only one child process is overridden at the same time. Because Redis itself is a CPU-intensive service, the child process is responsible for in-process data writes to the file in batches, consumes CPU, and the single core CPU will compete with the parent process to generate resources.
Memory:
Multiple Redis instances ensure that only one child process is working at the same time, and that child process overrides are avoided when a large number is written.
HDD:
Do not deploy on high-load hard drives; AoF do not do fsync operations during overrides (default off). AoF Additional blocking
Main thread->aof buffer, synchronous thread, synchronous disk, aof buffer---compare with last Fsync time, less than 2 seconds passed, more than 2 seconds blocked
Optimization of aof Append blocking problem is mainly to optimize the system hard disk load. Summary

The generated RDB is expensive and is generally used for data cold and replication transmission;
AoF file volume gradually become larger, periodically perform rewrite operations to reduce file volume;
AOF rewrite: auto-aof-rewrite-min-size,auto-aofrewritepercentage parameter automatic trigger, bgrewriteaof manual trigger;
During child process execution, Copy-on-write shares memory with the parent process, which doubles the memory consumption. AoF during rewrite, maintain rewrite buffers, save new write commands to avoid data loss;
Persistent blocking main thread: The fork blocking time is related to the memory and the system, aof additional blocking indicates that the hard disk resource is tight;
Single-machine multi-instance, anti-multiple sub-process rewrite, do isolation control.

Reference:
"Redis Development and operations"