Redis RDB Persistence

In the case of running, Redis saves data in memory as a structure, and in order for data to be available after Redis restarts, Redis provides two ways to persist the RDB and aof, this article explains how RDB persistence works

At run time, Redis stores the in-memory database to disk as a snapshot, and reads the RDB file on disk after a reboot to restore Redis to its original state advantages and disadvantages of the RDB

· An RDB is a very compact file that holds a Redis data set at a point in time and is ideal for backup

· The RDB is suitable for disaster recovery because it has only one file, and the content is compact and can be transferred to other data centers for saving

· An RDB maximizes the performance of Redis, only fork a subprocess when performing an RDB persistence, and persists with child processes, and the parent process does not need to process any disk I/O operations

· RDB is faster than aof when recovering large data sets

• An RDB is not appropriate if you try to avoid losing data when a server fails. Because the RDB persists to save data for the entire dataset, this is not an easy operation, so it is not suitable for frequent operations and may lose a few minutes of data once the server fails

• Each time an RDB is saved, the parent process will need to fork a child process for persistence, and if the dataset is very large, the time required for the fork can be very long, which could cause a request to pause processing the client

If the data set is huge and the CPU time is very tight, the stop time may even be a full second. Although the AOF rewrite also requires fork (), the durability of the data does not have any loss , regardless of how long the AOF rewrite is performed. Rdb Snapshot

By default, Redis saves a database snapshot in a binary file named Dump.rdb. You can set up Redis to automatically save a dataset once the condition that the dataset has at least M changes in N seconds is met. You can also manually let Redis do a dataset save operation by calling Save or Bgsave.
For example, the following settings will allow Redis to automatically save a dataset when it satisfies the condition "at least 1000 keys are changed in 60 seconds":
Save 60 1000
This persistence method is called snapshot (snapshot) format

An RDB file can be divided into the following sections:

Redis: A Redis 5 character is kept at the beginning of the file, indicating the beginning of the Rdb file

Rdb-version: A four-byte long integer that is represented by a character that records the RDB version number used by the file

Db-data: This section will appear more than once in an Rdb file, which holds all the data from the Redis non-empty database

Select-db: This field holds the number of the database to which the following key-value pairs belong

Key-value-pairs: Because an empty database is not saved to an RDB file, this part contains at least one key-value pair of data.

The data for each key-value pair is saved using the following structure:

Optional-expire-time domain is optional, if the key does not set the expiration time, then the domain will not appear, conversely, if the domain appears, then it records the key expiration time, in the current version of the RDB, the expiration time is a UNIX timestamp in milliseconds

The key field holds keys in the same format as Redis_encoding_raw encoded string objects
The Type-of-value field records the encoding used for values in the Value field, and depending on the field's instructions, the program uses different ways to save and read value

EOF: Indicates the end of the database content, edis_rdb_opcode_eof

Check-sum:rdb checksum of all contents of the file, one UINT_64T type value
REDIS officer and saves at the end of an RDB file when it is written to the Rdb file and, when read, validates the content against its value, the rdb Persistence command save

When the Save command executes, the Redis server is blocked, so when a save command executes, no additional save, Bgsave, or bgrewriteaof commands are processed

The new save, BGSAVE, or BGREWRITEAOF command will be processed only after the last save has been executed and Redis has started accepting requests again
In addition, because AOF writes are done by a background thread, and bgrewriteaof is done by a child process, AOF writes and bgrewriteaof can be BGSAVE simultaneously during save execution

Before executing the Save command, the server checks if BGSAVE is executing, and if so, the server does not call Rdbsave, but instead returns an error message to the client informing that the save is not executed during BGSAVE execution
This avoids the cross execution of SAVE and BGSAVE calls by two Rdbsave, creating a competitive condition

On the other hand, when BGSAVE is executing, the client calling the new BGSAVE command receives an error message that tells
BGSAVE is already in the middle of the execution.
Bgrewriteaof and BGSAVE cannot be executed at the same time:
• If BGSAVE is executing, the bgrewriteaof rewrite request is deferred until the BGSAVE execution completes, and the client executing the bgrewriteaof command receives a reply that the request was delayed.
• If bgrewriteaof is executing, the client calling BGSAVE will receive an error message indicating that the two commands cannot be executed at the same time.
Bgrewriteaof and BGSAVE Two commands do not have any conflict in the operational aspects, can not execute them at the same time is only a performance consideration: and issued two sub-processes, and two sub-processes are also a large number of disk write operations, it is not a good idea how to think Implementation of an RDB

The most central features of the RDB function are the rdbsave and rdbload two functions

The following specific analysis of the implementation of the Rdbsave function

Redis supports the RDB in two ways:

Current process execution and background execution (BGSAVE)

The RDB BGSAVE strategy is to fork out a sub-process and dump the data set in memory to the hard disk. Two scenarios for example Redis server initialization process, set a timed event, each time will trigger a persistent operation; into a timed event handler, the fork produces a child process to perform the persistence operation the Redis server presets the save instruction, and the client can request that the server process interrupt the service. To perform a persistence operation

The Rdbsave function is implemented as follows:

  1:/* Save the DB on disk. Return Redis_err On Error, REDIS_OK on success */

  2:/*

  3:  * Save the database to disk. Successfully returned REDIS_OK, failed to return Redis_err.

  4:  */

  5:int Rdbsave (char *filename) {

  6:     dictiterator *di = NULL;

  7:     dictentry *de;

  8:     Char tmpfile[256];

  9:     Char magic[10];

Ten:     Int J;

One:     long Long now = Mstime ();

:     FILE *FP;

:     Rio Rdb;

:     uint64_t cksum;

15: 

+     //create temporary file names in "Temp-<pid>.rdb" format

:     snprintf (tmpfile,256, "Temp-%d.rdb", (int) getpid ());

:     fp = fopen (Tmpfile, "w");

:     if (!FP) {

:         Redislog (redis_warning, "Failed opening. Rdb for saving:%s",

:             strerror (errno));

:         return redis_err;

:     }

24: 

:     //Initialize the Rio file

:     rioinitwithfile (&RDB,FP);

+/     /Set checksum calculation function if necessary

:     if (server.rdb_checksum)

£ º         rdb.update_cksum = Riogenericupdatechecksum;

+     //write to file header in "REDIS <VERSION>" format and version of RDB

To:     snprintf (Magic,sizeof (Magic), "redis%04d", redis_rdb_version);

:     if (Rdbwriteraw (&rdb,magic,9) = =-1) goto Werr;

33: 

:     //Traverse all databases to save their data

: For     (j = 0; J < Server.dbnum; J + +) {

#:         //point to Database

Panax Notoginseng:         redisdb *db = server.db+j;

:         //point to database key space

In:         dict *d = db->dict;

Max:         //The database is empty, pass, process the next database

:         if (dictsize (d) = = 0) continue;

42: 

:         //Create iterators

:         di = Dictgetsafeiterator (d);

:         if (!di) {

:             fclose (FP);

£ º             return redis_err;

:         }

49: 

:/         * Write the SELECT DB opcode */

Wuyi:         //Record the number of the database being used

:         if (rdbsavetype (&rdb,redis_rdb_opcode_selectdb) = =-1) goto Werr;

:         if (Rdbsavelen (&rdb,j) = =-1) goto Werr;

54: 

*/         * Iterate this DB writing every entry */

:         //Save all nodes in the database to an RDB file

Dictnext: While         ((de = (di)) = NULL) {

:             //Remove key

:             SDS keystr = Dictgetkey (DE);

Max:             //Remove value

A:             robj key, 

+:                  *o = Dictgetval (DE);

A:             long long expire;

64:             

:             initstaticstringobject (KEY,KEYSTR);

:             //Remove expiry time

:             expire = Getexpire (Db,&key);

:             if (Rdbsavekeyvaluepair (&rdb,&key,o,expire,now) = =-1) goto Werr;

A:         }

:         dictreleaseiterator (DI);

:     }

Again:     di = NULL;/* So, we don't release it on error. */

73: 

     * * * EOF opcode */

:     if (rdbsavetype (&rdb,redis_rdb_opcode_eof) = =-1) goto Werr;

76: 

*/     * CRC64 checksum. It'll be zero if checksum computation is disabled, the

      Loading code skips the check in this case. */

:     cksum = rdb.cksum;

:     memrev64ifbe (&cksum);

Bayi:     riowrite (&rdb,&cksum,8);

82: 

     * * * Make sure data won't remain on the OS ' s output buffers */

P:     fflush (FP);

:     Fsync (Fileno (FP));

P:     fclose (FP);

87: 

RENAME:/     * Use sure the DB file is changed atomically only

      * If the Generate DB file is OK. */

:     //Rename temporary file tmpfile to filename 

:     if (rename (tmpfile,filename) = =-1) {

Redis_warning:         redislog ("Error Moving temp DB file on the final destination:%s", Strerror (errno));

:         unlink (tmpfile);

94:         return Redis_err;

:     }

96: 

:     Redislog (Redis_notice, "DB saved on disk");

98: 

:     //Initialize database data

:     server.dirty = 0;

101:     Server.lastsave = time (NULL);

102:     server.lastbgsave_status = REDIS_OK;

103: 

104:     return REDIS_OK;

105: 

106:werr:

107:     fclose (FP);

108:     unlink (tmpfile);

109:     Redislog (redis_warning, "Write Error saving DB on disk:%s", Strerror (errno));

:     if (di) dictreleaseiterator (DI);

111:     return redis_err;

112:}

In Rdbsave (), create a temporary file, rename it after successful save, then initialize Rio (robust IO, which is the wrapper for IO operations, dedicated to DDB save in Redis), and then record the Redis MAGIC number first in the file. The rdb_checksum here is the checksum, which updates the checksum whenever it writes to the hard disk, for verifying when the Rdb file is later read

Dictgetsafeiterator () is an iterator to the DICT data structure that iterates through the database in Redis instance, first writing the REDIS_RDB_OPCODE_SELECTDB prefix and the DB number. The Rdbsavelen () here is a function that implements the length compression encoding, saving the length with 6bits, 14bits, or 32bits. It then traverses the db to get the key and value, and possibly the expire time. Because at the DB level, key is a string of the ads structure, which needs to be wrapped as a RobJ object and then saved as a string type. Call Rdbsavekeyvaluepair () to resolve the value type, save the format according to the type

Finally write the checksum, synchronize the contents to the hard disk, and then rename the file

The following is a bgsave,fork generation of a child process

  1: 

  2:/*

  3:  * Use child process to save database data without blocking main process

  4:  */

  5:int Rdbsavebackground (char *filename) {

  6:     pid_t childpid;

  7:     long long start;

  8: 

  9:     if (server.rdb_child_pid! =-1) return redis_err;

10:     

One:     //Modify Server Status

:     server.dirty_before_bgsave = Server.dirty;

13: 

:     //Start time

:     start = Ustime ();

:     //Create child process

:     if ((Childpid = fork ()) = = 0) {

:         int retval;

19: 

*/         * Child */

:         //child processes do not receive network data

:         if (server.ipfd > 0) Close (SERVER.IPFD);

:         if (server.sofd > 0) Close (SERVER.SOFD);

24: 

:         //Save data

:         retval = rdbsave (filename);

:         if (retval = = REDIS_OK) {

:             size_t private_dirty = Zmalloc_get_private_dirty ();

29: 

:             if (private_dirty) {

To:                 Redislog (Redis_notice,

:                     "RDB:%lu MB of memory used by Copy-on-write",

<