Redis persistence, master-Slave and data backup

http://blog.csdn.net/lang_man_xing/article/details/38386113

Redis has been used extensively in projects now, and to improve the performance and reliability of Redis we need to know and do the following:

Common memory optimization methods and parameters

The performance of Redis is completely dependent on memory, so we need to know how to control and save memory.

First, the most important thing is not to turn on the redis VM option, the virtual Memory feature, which is a persistent policy that is used as a redis storage out of physical memory data in memory and disk swap, but its memory management costs are very high, so to turn off the VM function, Please check your redis.conf file for vm-enabled No.

Next, it is best to set the MaxMemory option in Redis.conf, which tells Redis how much physical memory is used to start rejecting subsequent write requests, which is a good way to protect your redis from using too much physical memory to cause swap. Eventually seriously impacting performance or even crashing.

In addition, Redis provides a set of parameters for different data types to control memory usage, and we know that Redis hash is a hashmap within value, and if the map has a smaller number of members, it will be stored in a compact format similar to one-dimensional linear, This eliminates the memory overhead of a large number of pointers, and this parameter controls the following 2 entries in the redis.conf configuration file:

Hash-max-zipmap-entries 64
Hash-max-zipmap-value 512

The implication is that when value is not more than the number of members within the map is stored in a linear compact format, the default is 64, that is, the value of 64 members of the following is the use of linear compact storage, more than this value automatically into a real hashmap.

Hash-max-zipmap-value means that when value does not exceed the number of bytes per member within the map, linear compact storage is used to save space.

Above 2 conditions any one condition exceeds the set value to convert to the real HashMap, also will not save the memory again, then this value is not set the bigger the better, the answer is of course negative, the HashMap advantage is to find and operate the time complexity is O (1), Discard hash using one-dimensional storage is O (n) time complexity, if the number of members is small, the impact is small, otherwise it will seriously affect performance, so to weigh the value of the setting, overall is the most fundamental time cost and space cost tradeoff.

Also similar parameters are:

List-max-ziplist-entries 512

Description: The list data type is a compact storage format that uses a pointer to the following number of nodes.

List-max-ziplist-value 64

Description: The list data type node value size is less than how many bytes are in a compact storage format.

Set-max-intset-entries 512

Description: The Set data type internal data is stored in a compact format if it is all numeric and contains many nodes.

The Redis internal implementation does not optimize memory allocations to some extent, but in most cases this does not become a performance bottleneck for Redis, but if most of the data stored inside Redis is numeric, the Redis internal uses a shared Integer way to eliminate the overhead of allocating memory, that is, when the system starts to allocate a number of numeric objects from the 1~n in a pool, if the stored data is exactly the value of the data in the range, then directly from the pool to remove the object, and by reference to count the way to share, This allows the system to store a large number of values, but also to a certain extent save memory and improve performance, this parameter value n settings need to modify the source code in a row of macro definition redis_shared_integers, the value is 10000 by default, can be modified according to their own needs, After the modification, you can recompile.

Persistence of

Redis is an in-memory database that supports persistence, which means that Redis often needs to synchronize in-memory data to disk to ensure persistence. Redis supports two persistence modes, one is snapshotting (snapshot) is the default, and the other is the way Append-only file (abbreviated AOF).

Snapshotting

Snapshots are the default persistence mode. This is the way in which the in-memory data is written to the binary in a snapshot, the default file name is Dump.rdb. You can automatically make snapshot persistence by configuring settings. We can configure Redis to take snapshots automatically if more than M key is modified in n seconds, the following is the default snapshot save configuration:

[Plain]View Plaincopy

1. Save 1 #If more than 1 key is modified within 900 seconds to initiate a snapshot save
   Save # 300 seconds if more than 10 keys are modified, the snapshot save is initiated
   Save 10000 # 60 seconds content is modified if more than 10,000 keys, initiate a snapshot save

You can also make Redis snapshotting in the command line mode:

[Plain]View Plaincopy

1. Redis-cli-h ip-p Port Bgsave

Save the snapshot there are save and bgsave two commands, save operation is saved in the main thread of the snapshot, because Redis is a main thread to handle all client requests, this way will block all client requests, it is not recommended to use.

The snapshot generation process is roughly as follows:

1. Redis calls fork and now has child and parent processes;
2. The parent process continues to process the client request, and the child process is responsible for writing the memory contents to the temporary file. Because the OS's write-time replication mechanism (copy on write) will share the same physical page, when the parent process processes the write request, the OS creates a copy of the page to be modified by the parent process, rather than writing the shared page. So the data in the address space of the child process is a snapshot of the entire database at fork time;
3. When the child process finishes writing the snapshot to the temporary file, replace the original snapshot file with a temporary file, and the child process exits.

At the same time snapshotting is also insufficient, because there is a time interval between the snapshot operations, once the database has a problem, the snapshot file saved data is not entirely new, from the last snapshot file generation to Redis outage time of the data are all discarded. If the business is very demanding on data accuracy, the AOF persistence mechanism has to be adopted.

AoF

AOF is more persistent than snapshot mode, because Redis appends each received write command to a file by using the Write function (default is appendonly.aof) when aof persistence is used. When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands saved in the file. Of course, because the OS caches write modifications in the kernel, it may not be written to disk immediately. The persistence of this aof method is also likely to lose some of the modifications. But we can tell Redis through the configuration file that we want to force the OS to write to disk through the Fsync function. There are three ways to do this (the default is: Fsync once per second):

[Plain]View Plaincopy

1. AppendOnly Yes//enable AOF persistence mode
2. # Appendfsync always//write command is immediately forced to write to disk, the slowest, but guaranteed full persistence, not recommended
3. Appendfsync everysec//force write disk once per second, a good compromise in performance and persistence, recommended
4. # Appendfsync no//completely dependent on OS, best performance, no guarantee of persistence

The AOF approach also poses another problem. Persistent files can become more and more large. For example, we call the INCR Test command 100 times, the file must save all 100 commands, in fact, 99 are redundant. Because you want to restore the state of the database, it is enough to save a set test 100 in the file. In order to compress the aof persistence file. Redis provides the bgrewriteaof command. Receive this command Redis will use a snapshot-like method to save the in-memory data to a temporary file in the form of a command, and finally replace the original file. The bgrewriteaof command is as follows:

[Plain]View Plaincopy

1. Redis-cli-h ip-p Port Bgrewriteaof

The BGREWRITEAOF command executes the following procedure:

1. Redis calls Fork, now there are two processes of father and son;
2. The child process writes to the temporary file the command to rebuild the database state based on the database snapshot in memory;
3. The parent process continues to process the client request, in addition to writing the write command to the original aof file. Cache the received write commands at the same time. This will ensure that if the sub-process rewrite fails, it will not cause problems;
4. When a child process writes the snapshot content to a temporary file as a command, the child process signals the parent process. The parent process then writes the cached write command to the temporary file as well;
5. Now the parent process can replace the old aof file with the temporary file and rename it, and the subsequent write commands are appended to the new AoF file.

The two persistence modes have their own characteristics, fast photography has little impact on performance, but once the crash, the data loss is large, and the AOF data security is high, but the performance impact is large, this has to be based on the business characteristics of their own choice.

Master-slave replication

The master-slave replication policy of Redis is implemented by its persisted RDB file, which is the process of dump the Rdb file, transfer the Rdb file to slave, and then synchronize the operation of the dump to slave in real time.

To use the master-slave feature requires a simple configuration on the slave side:

[Plain]View Plaincopy

1. slaveof master_ip master_port #如果这台机器是台redis Slave, you can open this setting.
2. Slave-serve-stale-data no #如果slave cannot sync with master and is set to slave unreadable for easy monitoring of script discovery issues.

After the configuration is started, the slave end can be replicated, the master-slave replication process is generally as follows:

1. Slave end in the configuration file added slaveof instructions, so slave read the configuration file at startup, the initial state is redis_repl_connect;
2. The slave end connects master in the timed task Servercron (Timer trigger event within Redis), sends the Sync command, and then blocks waiting for master to send back its memory snapshot file (the latest version of Redis does not need to let slave block);
3. The master side receives the sync command to simply determine if there is an ongoing memory snapshot subprocess, does not immediately start a memory snapshot, there is a wait for it to end, when the snapshot is completed, the file will be sent to the slave side;
4. The slave side receives the memory snapshot file from master, saves it locally, clears the memory table when it is received, re-reads the memory snapshot file from master, rebuilds the entire memory table data structure, and the final state is set to the redis_repl_connected state. Slave state machine flow is completed;
5. The master side in the process of sending the snapshot file, any command that changes the data set will be temporarily saved in the slave network connection of the Send cache queue joins (list data structure), after the snapshot is completed, sent to slave, followed by the same processing of the command received, and the status is set to Redis_ Repl_online.

The entire replication process is complete, as shown in the process:

From the above replication process can be found, slave from the library when connected to the Master Master Library, Master will take a memory snapshot, and then send the entire snapshot file to slave, that is, there is no replication location like MySQL concept, that is, no incremental replication, If a master is connected to multiple slave, it will affect master performance more.

Data backup Strategy

The specific backup strategy can be very flexible, for example, can be roughly as follows:

1. In order to improve the performance of Master shutdown Master Persistence mechanism, that is, do not take a snapshot or aof, but in the early morning when the traffic is low, timed to use the Bgsave command snapshot, and save the snapshot file to the backup server;
2. Slave-side open aof mechanism, and the timing of the data compression with the bgrewriteaof, the compressed data files are saved to the backup server;
3. Check whether the data on master and slave is consistent on a regular basis;
4. When Master has problems and needs to be recovered, If a backup snapshot of master is restored directly to the DUMP.RDB copy of the backup to the appropriate path, if you want to recover from the slave side, you need to perform a snapshot on the slave side, then copy the snapshot files to the master path and then reboot. One thing to note, however, is that the slave data is flushed out when the master restarts, so the slave end should be backed up before the master restarts.

Persistent disk IO mode and the problems it brings

People who have experience with Redis on-line operations will find that Redis has a lot of physical memory usage, but it has not exceeded the actual physical memory of the total capacity of the instability or even crash, some people think that is based on the snapshot persistence of the fork system calls resulting in double memory consumption, this view is inaccurate, Because the copy-on-write mechanism of the fork call is based on the unit of the operating system page, that is, only dirty pages that have been written are copied, but the general system does not write to all pages in a short period of time, resulting in replication, what causes Redis to crash?

The answer is that the persistence of Redis uses buffer Io, which means that Redis writes and reads to persisted files using the page Cache of physical memory, and most database systems use direct IO to bypass this layer of page Cache and maintain a cache of the data itself, and when a Redis persistent file is too large (especially a snapshot file) and read and write to it, the data in the disk file is loaded into physical memory as the operating system caches a layer of the file. This cache data and Redis memory management of the actual data is repeatedly stored, although the kernel in the physical memory when the page cache does not work, but the kernel may think that a piece of page cache more important, and let your process start swap, Your system will start to appear unstable or crash. The experience is that when your Redis physical memory is using more than 3/5 of the total memory capacity, it starts to be more dangerous.

1. Snapshot persisted to diskAutomatic persistence rule configuration save 1save 10save 60 10000 The above configuration rules mean the following: # In the example below the behaviour would be and save:# after 90 0 sec If at least 1 key changed# after the SEC (5 min) If at least the keys changed# after ten sec if at least 10000 keys Changedredis can also turn off automatic persistence, comment out these save configurations, or save "" If a background save to disk error occurs, the write operation will stop. Stop-writes-on-bgsave-error Yes use LZF to compress the Rdb file, which consumes the CPU, but can reduce disk usage. rdbcompression Yes to save the RDB and load the Rdb file when the test can prevent errors, but to pay about 10% performance, you can turn off him, improve performance. Rdbchecksum Yes to export the Rdb file name Dbfilename Dump.rdb set working directory, RDB files will be written to that directory, append only file will also be stored in the directory. Dir./ Redis automatic snapshots are saved to disk or called Bgsave, which is done by the background process, and other clients can still read and write to the Redis server, and the background to save the snapshot to disk consumes a lot of memory. Calling save to save the in-memory data to disk will block the client request until the save is complete. Calling the shutdown command, the Redis server calls save first, and all data is persisted to disk before it actually exits. For data loss issues:

If the server crash, the data from the last snapshot will be lost. So when you set the save rule, you want to set the allowable range based on the actual business settings. If the data is sensitive to the business, in the program to use the appropriate log, after the server crash, data recovery through the log.

2. append-only File Mode Persistence Another way is to increment the way that will cause data to change the operation, persist to the file, restart Redis, by Operation Command, restore the data. Each time the write command is executed, the data is written to Server.aofbuf. # Appendfsync Alwaysappendfsync everysec# Appendfsync No when configured as always, each time the data in the SERVER.AOFBUF is written to the file, it is returned to the client. This ensures that the data is not lost, but frequent IO operations can degrade performance.  Everysec writes once per second, which can lose operations within one second. AoF The biggest problem is that as time append file will become very large, so we need to bgrewriteaof command to reorganize the file, only the latest KV data.

Redis persistence, master-Slave and data backup