Redis article Redis persistence mode aof and RDB

First, the foregoing

Persistence concept: Store data from lost and volatile memory on devices that can be permanently stored.

Redis Persistence mode
RDB (Redis DB) hdfs:fsimage
AOF (appendonlyfile) Hdfs:edit logs off by default

Second, the RDB mode

By default, Redis saves a db snapshot in a binary file named Dump.rdb

In the RDB mode, there are two ways to

1, one is to manually execute persistent data command to let Redis take a snapshot of the data, and manually execute the persistence command, you still have two options, that is the save Command and Bgsave command.

Save

The client executes the Save command manually
Redis > Save
blocking Redis service , unable to respond to client requests
Create new Dump.rdb to replace old files

Bgsave: is an asynchronous command
Redis > Bgsave
Non-blocking, Redis service gracefully receives processing client requests
Redis will fork () a new child process to create an RDB file, and the child process will send a signal to the parent process to notify it that it has finished processing
Parent process replaces old files with new Dump.rdb

Attention:

When fork occurs, the parent-child process memory is shared, so in order to not affect the child process to do the data snapshot, during this time the modified data will be copied one copy, not into the shared memory . So, thedata persisted by R DB is the data when the fork occurs . Persist data under such conditions and lose data for a period of time if it is down due to certain circumstances . If your situation is less sensitive to data loss, you can get lost from a traditional database or lose part of it, then you can choose the Rdb persistence method.

Comparison:

SAVE and BGSAVE commands
Save does not create a new process, slightly faster
Bgsave need to create child processes and consume additional memory
Save for downtime maintenance, service trough time
Bgsave for online execution

2, the other is based on the configuration file you configured policies , to achieve certain conditions of the policy to automatically persist data. same principle as Bgsave execution

This is the default policy for profiles, and the relationship between them is or, every second of every minute, changes at least one key value during this time, making a snapshot. Or every 300 seconds , 10 key values are changed to take a snapshot. or every 60 seconds, change at least 10,000 key values, take a snapshot.

Three, aof way

Ppend only file, saved in Append mode
Default File Appendonly.aof
log all write commands and use these commands to restore the database when the service is started

Adjust the AOF persistence policy to not lose any data in the event of a service failure, or to lose one second of data. Much less loss relative to Rdb

1, aof write mechanism ( but in fact, does not immediately write the command to the hard disk file, but write to the hard disk cache, in the next policy, configure how long to write from the hard disk cache to the hard disk files. So under certain conditions, there will still be data loss, but you can greatly reduce the loss of data. )
AOF mode does not guarantee absolute loss of data
In the current operating system, the system calls the Write function, writes some content to a file, in order to improve efficiency, the system usually does not directly write the content to the hard disk, but first put the content into a memory buffer (in buffer), and then until the buffers are filled, Or when the user executes the Fsync call and the Fdatasync call, the content stored in the buffer is actually written to the hard disk, and the data may be lost before it is written to the disk.

2. Write to disk policy ( here is the policy for configuring AOF persistence. Redis uses everysecby default, which means that it is persisted once per second, and always is written to the AoF file immediately per operation. And no is not active synchronous operation, is the default of 30s a time. of course always must be the lowest efficiency,everysec is enough, the data security can be high. )
Appendfsync option, the value of this option can be always, everysec, or no
Always: Each time the server writes a command, it invokes a fdatasync and writes the command inside the buffer to the hard disk. In this mode, the server fails without losing any command data that has been successfully executed
Everysec (default): The server calls the Fdatasync once every second and writes the commands inside the buffer to the hard disk. In this mode, the server fails with a maximum of one second of executed command data
No: The server does not actively invoke Fdatasync, which is determined by the operating system when the commands inside the buffer are written to the hard disk. In this mode, the number of LOST commands is indeterminate when the server encounters an unplanned outage
Running speed: Always slow, everysec and no are fast

3. AOF rewrite mechanism

AOF the command operations of Redis in an orderly manner. There is little data loss in the event of an accident. He keeps adding operational logging to the AoF file, and you might say how big the file is. Yes, it does get huge, but Redis has an optimized strategy, such as your operation on a Key1 key, set Key1 001, set Key1 002, set Key1 003. The result of the optimization is to remove the first two, the configuration of the specific optimization in the configuration file corresponding to the

The former refers to how much more than the last aof rewrite aof file size, will be optimized again, if not rewritten, then start with the main. The latter is limited to the minimum aof file size allowed for rewriting. bgrewriteaof command is to manually rewrite the command, will fork the child process, in the temporary file to rebuild the state of the database, the original aof has no effect, when the old state is rebuilt, the fork after the occurrence of a period of time data is appended to the temporary file, and finally replace the original aof file, The new command continues to append to the new aof file.

AOF file is too large
Merging duplicate operations, AOF uses as few commands as possible to log
Rewrite process
Fork a child process is responsible for overriding the AoF file
Child processes create a temporary file to write aof information
The parent process will open up a memory buffer to receive the new write command
When the child process rewrite is complete, the parent process obtains a signal that the parent process will receive a new write operation that is written to the temporary file by the descendant process
New file replaces old file
Note: If the write operation fails to cause the command to write half, you can use the Redis-check-aof tool to repair

AOF Override Trigger
Manual: The client sends the BGREWRITEAOF command to the server
Auto: option in config file, automatic execution of bgrewriteaof command
Auto-aof-rewrite-min-size <size>, the minimum volume required to trigger the AOF override: Whenever the AoF file's volume is greater than or equal to size, the need for AOF overrides is considered. This option is used to avoid overwriting aof files that are too small in size.
Auto-aof-rewrite-percentage <percent> Specifies the percentage of the AoF file volume that is required to trigger the override: when the volume of the aof file is greater than the volume specified by Auto-aof-rewrite-min-size, And the AOF override is triggered when the percent% of the aof file volume after the last rewrite is exceeded. (If the server has just started and has not yet been aof rewritten, use the volume of the AoF file loaded at server startup as the baseline value). Setting this value to 0 means turning off automatic aof overrides

AoF overriding Configuration Items Example
Auto-aof-rewrite-percentage 100
Auto-aof-rewrite-min-size 64MB
AppendOnly No/yes
When the AoF file is larger than 64MB, you can consider rewriting the aof file
Start rewrite only if the increment of the aof file is greater than 100% of the starting size (that is, the file size is doubled)
Off by default, turn on

Iv. comparison of Rdb and AOF

1. RDB:

Advantages:
Full backup, multi-version recovery with data set backups at different times
Compact single file for easy network transmission for disaster recovery
Recover large data sets faster than AOF
Disadvantages:
Data that is recently written, modified, and not persisted is lost
The fork process is time consuming and can cause the millisecond level to not respond to client requests

2, AOF

Advantages
Write mechanism, default FYSNC per second execution, performance is not good blocking service, up to one second of data loss
Rewrite mechanism to optimize AOF files
If the operation is wrong (flushall, etc.), as long as aof is not overridden, the stop service removes the aof file Tail flushall command, and restarts Redis to restore the dataset to the state before Flushall execution
Disadvantages
The same data set, the AoF file volume is much larger than the RDB
Recover database slower than RDB (text, command replay)

Redis article Redis persistence mode aof and RDB