Redis memory optimization, persistence, and master-slave replication

the configuration of Redis database memory optimization parameters, the pros and cons of each persistence method, and the principle and configuration of master-slave replicationfirst, commonly used memory optimization methods and parameters

The performance of Redis is completely dependent on memory, so we need to know how to control and save memory.

First, the most important thing is not to turn on the redis VM option, the virtual Memory feature, which is a persistent policy that is used as a redis storage out of physical memory data in memory and disk swap, but its memory management costs are very high, so to turn off the VM function, Please check your redis.conf file for vm-enabled No.

Next, it is best to set the MaxMemory option in Redis.conf, which tells Redis how much physical memory is used to start rejecting subsequent write requests, which is a good way to protect your redis from using too much physical memory to cause swap. Eventually seriously impacting performance or even crashing.

In addition, Redis provides a set of parameters for different data types to control memory usage, and we know that Redishash is a hashmap within value, and if the map has fewer members, it will be stored in a compact format similar to one-dimensional linearity, This eliminates the memory overhead of a large number of pointers, and this parameter controls the following 2 entries in the redis.conf configuration file:

Hash-max-zipmap-entries 64

Hash-max-zipmap-value 512

The implication is that when value is not more than the number of members within the map is stored in a linear compact format, the default is 64, that is, the value of 64 members of the following is the use of linear compact storage, more than this value automatically into a real hashmap.

Hash-max-zipmap-value means that when value does not exceed the number of bytes per member within the map, linear compact storage is used to save space.

Above 2 conditions any one condition exceeds the set value to convert to the real HashMap, also will not save the memory again, then this value is not set the bigger the better, the answer is of course negative, the HashMap advantage is to find and operate the time complexity is O (1), Discard hash using one-dimensional storage is O (n) time complexity, if the number of members is small, the impact is small, otherwise it will seriously affect performance, so to weigh the value of the setting, overall is the most fundamental time cost and space cost tradeoff.

Also similar parameters are:

List-max-ziplist-entries 512

Description: The list data type is a compact storage format that uses a pointer to the following number of nodes.

List-max-ziplist-value 64

Description: The list data type node value size is less than how many bytes are in a compact storage format.

Set-max-intset-entries 512

Description: The Set data type internal data is stored in a compact format if it is all numeric and contains many nodes.

The Redis internal implementation does not optimize memory allocations to some extent, but in most cases this does not become a performance bottleneck for Redis, but if most of the data stored inside Redis is numeric, the Redis internal uses a Sharedinteger way to eliminate the overhead of allocating memory, that is, when the system starts to allocate a number of 1~n from the number of objects placed in a pool, if the stored data is exactly the value of the data in the range, then directly from the pool to remove the object, and by reference to count the way to share, This allows the system to store a large number of values, but also to a certain extent save memory and improve performance, this parameter value n settings need to modify the source code in a row of macro definition redis_shared_integers, the value is 10000 by default, can be modified according to their own needs, After the modification, you can recompile.

second, the persistence of

Redis is an in-memory database that supports persistence, which means that Redis often needs to synchronize in-memory data to disk to ensure persistence. Redis supports two persistence modes, one is snapshotting (snapshot) is the default, and the other is the way Append-only file (abbreviated AOF).

Snapshotting

Snapshots are the default persistence mode. This is the way in which the in-memory data is written to the binary in a snapshot, the default file name is Dump.rdb. You can automatically make snapshot persistence by configuring settings. We can configure Redis to take snapshots automatically if more than M key is modified in n seconds, the following is the default snapshot save configuration:

1. Save 1 #900秒内如果超过1个key被修改 to initiate a snapshot save
2. Save #300秒内容如超过10个key被修改, the snapshot save is initiated
3. Save 10000 #60秒内容如超过10000个key被修改, initiate a snapshot save

You can also make Redis snapshotting in the command line mode:

1. Redis-cli-h ip-p Port Bgsave

Save the snapshot there are save and bgsave two commands, save operation is saved in the main thread of the snapshot, because Redis is a main thread to handle all client requests, this way will block all client requests, it is not recommended to use.

The snapshot generation process is roughly as follows:

1. Redis calls fork and now has child and parent processes;
2. The parent process continues to process the client request, and the child process is responsible for writing the memory contents to the temporary file. Because the OS's write-time replication mechanism (copy on write) will share the same physical page, when the parent process processes the write request, the OS creates a copy of the page to be modified by the parent process, rather than writing the shared page. So the data in the address space of the child process is a snapshot of the entire database at fork time;
3. When the child process finishes writing the snapshot to the temporary file, replace the original snapshot file with a temporary file, and the child process exits.

At the same time snapshotting is also insufficient, because there is a time interval between the snapshot operations, once the database has a problem, the snapshot file saved data is not entirely new, from the last snapshot file generation to Redis outage time of the data are all discarded. If the business is very demanding on data accuracy, the AOF persistence mechanism has to be adopted.

AoF

AOF is more persistent than snapshot mode, because Redis appends each received write command to a file by using the Write function (default is appendonly.aof) when aof persistence is used. When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands saved in the file. Of course, because the OS caches write modifications in the kernel, it may not be written to disk immediately. The persistence of this aof method is also likely to lose some of the modifications. But we can tell Redis through the configuration file that we want to force the OS to write to disk through the Fsync function. There are three ways to do this (the default is: Fsync once per second):

1. AppendOnly Yes//enable AOF persistence mode
2. # Appendfsync always//write command is immediately forced to write to disk, the slowest, but guaranteed full persistence, not recommended
3. Appendfsync everysec//force write disk once per second, a good compromise in performance and persistence, recommended
4. # Appendfsync no//completely dependent on OS, best performance, no guarantee of persistence

The AOF approach also poses another problem. Persistent files can become more and more large. For example, we call the Incrtest command 100 times, the file must be saved all 100 commands, in fact, 99 are redundant. Because to restore the state of the database, it is sufficient to save a set test100 in the file. In order to compress the aof persistence file. Redis provides the bgrewriteaof command. Receive this command Redis will use a snapshot-like method to save the in-memory data to a temporary file in the form of a command, and finally replace the original file. The bgrewriteaof command is as follows:

1. Redis-cli-h ip-p Port Bgrewriteaof

The BGREWRITEAOF command executes the following procedure:

1. Redis calls Fork, now there are two processes of father and son;
2. The child process writes to the temporary file the command to rebuild the database state based on the database snapshot in memory;
3. The parent process continues to process the client request, in addition to writing the write command to the original aof file. Cache the received write commands at the same time. This will ensure that if the sub-process rewrite fails, it will not cause problems;
4. When a child process writes the snapshot content to a temporary file as a command, the child process signals the parent process. The parent process then writes the cached write command to the temporary file as well;
5. Now the parent process can replace the old aof file with the temporary file and rename it, and the subsequent write commands are appended to the new AoF file.

The two persistence modes have their own characteristics, fast photography has little impact on performance, but once the crash, the data loss is large, and the AOF data security is high, but the performance impact is large, this has to be based on the business characteristics of their own choice.

third, master-slave replication

The master-slave replication policy of Redis is implemented by its persisted RDB file, which is the process of dump the Rdb file, transfer the Rdb file to slave, and then synchronize the operation of the dump to slave in real time.

To use the master-slave feature requires a simple configuration on the slave side:

1. slaveof master_ip master_port #如果这台机器是台redis Slave, you can open this setting.
2. Slave-serve-stale-data no #如果slave cannot be synchronized with master and is set to slave unreadable for easy monitoring of script discovery problems.
3. If Master--slave (master)--slave This mode is the same as the master--slave this pattern is the same, so the configuration file and is the same as the Master--slave configuration can be.

After the configuration is started, the slave end can be replicated, the master-slave replication process is generally as follows:

1. Slave end in the configuration file added slaveof instructions, so slave read the configuration file at startup, the initial state is redis_repl_connect;
2. The slave end connects master in the timed task Servercron (Timer trigger event within Redis), sends the Sync command, and then blocks waiting for master to send back its memory snapshot file (the latest version of Redis does not need to let slave block);
3. The master side receives the sync command to simply determine if there is an ongoing memory snapshot subprocess, does not immediately start a memory snapshot, there is a wait for it to end, when the snapshot is completed, the file will be sent to the slave side;
4. The slave side receives the memory snapshot file from master, saves it locally, clears the memory table when it is received, re-reads the memory snapshot file from master, rebuilds the entire memory table data structure, and the final state is set to the redis_repl_connected state. Slave state machine flow is completed;
5. The master side in the process of sending the snapshot file, any command that changes the data set will be temporarily saved in the slave network connection of the Send cache queue joins (list data structure), after the snapshot is completed, sent to slave, followed by the same processing of the command received, and the status is set to Redis_ Repl_online.

The entire replication process is complete, as shown in the process:

From the above replication process can be found, slave from the library when connected to the Master Master Library, Master will take a memory snapshot, and then send the entire snapshot file to slave, that is, there is no replication location like MySQL concept, that is, no incremental replication, If a master is connected to multiple slave, it will affect master performance more.

Iv. Data backup Strategy

The specific backup strategy can be very flexible, for example, can be roughly as follows:

1. In order to improve the performance of Master shutdown Master Persistence mechanism, that is, do not take a snapshot or aof, but in the early morning when the traffic is low, timed to use the Bgsave command snapshot, and save the snapshot file to the backup server;
2. Slave-side open aof mechanism, and the timing of the data compression with the bgrewriteaof, the compressed data files are saved to the backup server;
3. Check whether the data on master and slave is consistent on a regular basis;
4. If Master has a problem and needs to recover, if the backup snapshot of master is restored directly to the DUMP.RDB copy to the corresponding path, a snapshot will need to be performed on the slave side if it is to be recovered from the slave side. Then copy the snapshot files to the master path and reboot. One thing to note, however, is that the slave data is flushed out when the master restarts, so the slave end should be backed up before the master restarts.

Persistent disk IO mode and the problems it brings

People who have experience with Redis on-line operations will find that Redis has a lot of physical memory usage, but it has not exceeded the actual physical memory of the total capacity of the instability or even crash, some people think that is based on the snapshot persistence of the fork system calls resulting in double memory consumption, this view is inaccurate, Because the copy-on-write mechanism of the fork call is based on the unit of the operating system page, that is, only dirty pages that have been written are copied, but the general system does not write to all pages in a short period of time, resulting in replication, what causes Redis to crash?

The answer is that Redis's persistence is caused by Bufferio, which is called buffer Io, which uses the physical memory of the page Cache for the Redis write and read operations on persisted files. Most database systems use Directio to bypass this layer of pagecache and maintain a cache of the data themselves, and when Redis persistent files are too large (especially for snapshot files) and read and written, The data in the disk file is loaded into physical memory as a layer of cache for the operating system to the file, and the cache data is actually stored repeatedly with the data in Redis memory, although the kernel does Pagecache when physical memory is tight. But the kernel may think that a piece of pagecache is more important, and let your process start swap, when your system will start to appear unstable or crash. The experience is that when your Redis physical memory is using more than 3/5 of the total memory capacity, it starts to be more dangerous.

1. Snapshot persisted to disk

Automatic Persistence rule configuration

Save 900 1

Save 300 10

Save 60 10000

The above configuration rules mean the following:

# in the example below the behaviour would be is to save:

# After the SEC (min) If at least 1 key changed

# After the SEC (5 min) If at least ten keys changed

# after the SEC if at least 10000 keys changed

Redis can also turn off automatic persistence, comment out these save configurations, or save ""

If there is an error in the background save to disk, the write operation will stop.

Stop-writes-on-bgsave-erroryes

Compressing an RDB file with Lzf consumes the CPU, but can reduce disk usage.

Rdbcompression Yes

When you save an RDB and load an RDB file, you can prevent errors, but with about 10% performance, you can turn them off and improve performance.

Rdbchecksum Yes

Exported RDB file name

Dbfilename Dump.rdb

Set the working directory, the Rdb file will be written to that directory, and append only file will be stored in that directory.

Dir./

Redis automatic snapshots are saved to disk or called Bgsave, which is done by the background process, and other clients can still read and write to the Redis server, and the background to save the snapshot to disk consumes a lot of memory. Calling save to save the in-memory data to disk will block the client request until the save is complete.

Calling the shutdown command, the Redis server calls save first, and all data is persisted to disk before it actually exits.

For data loss issues:

If the server crash, the data from the last snapshot will be lost. So when you set the save rule, you want to set the allowable range based on the actual business settings.

If the data is sensitive to the business, in the program to use the appropriate log, after the server crash, data recovery through the log.

2. append-only File Mode persistence

Another way is to increment the way that will cause data to change the operation, persist to the file, restart Redis, by Operation Command, restore the data.

Each time the write command is executed, the data is written to Server.aofbuf.

# Appendfsync Always

Appendfsync everysec

# Appendfsync No

When configured as always, each time the data in the SERVER.AOFBUF is written to a file, it is returned to the client, which guarantees that the data is not lost, but frequent IO operations can degrade performance.

Everysec writes once per second, which can lose operations within one second.

AoF The biggest problem is that as time AppendFile will become very large, so we need to bgrewriteaof order to reorganize the file, only the latest KV data.

Redis memory optimization, persistence, and master-slave replication