Redis Optimization Experience Summary

On the memory management optimization, this article mainly share with you the Redis Optimization experience summary, hope to help everyone.

The Redis hash is a hashmap within value, and if the map has fewer members, the map is stored in a compact format similar to one-dimensional linearity, which eliminates the memory overhead of a large number of pointers, which control the following 2 entries in the redis.conf configuration file:

Hash-max-zipmap-entries Hash-max-zipmap-value 512

When value is not more than the number of members within the map is stored in a linear compact format, the default is 64, that is, the value of 64 members of the following is the use of linear compact storage, more than this value automatically into a real hashmap.

Hash-max-zipmap-value means that when value does not exceed the number of bytes per member within the map, linear compact storage is used to save space.

Above 2 conditions any one condition exceeds the set value to convert to the real HashMap, also will not save the memory again, then this value is not set the bigger the better, the answer is of course negative, the HashMap advantage is to find and operate the time complexity is O (1), While discarding a hash using one-dimensional storage is the time complexity of O (n), if

The number of members is small, the impact is not significant, otherwise it will seriously affect performance, so to weigh this value of the setting, overall is the most fundamental time cost and space cost tradeoff.

List-max-ziplist-value List-max-ziplist-entries 512

The list data type node value size is less than how many bytes will be in compact storage format, list data type how many nodes below will use a compact storage format for pointers.

Memory Pre-allocation:

The Redis internal implementation does not optimize memory allocations (compared to memcache) to some extent with memory fragmentation, but in most cases this does not become a performance bottleneck for Redis, but if most of the data stored inside Redis is numeric, The Redis interior employs a shared integer to eliminate the overhead of allocating memory, which is to assign a number of numeric objects from 1~n to a pool when the system starts, and if the stored data happens to be data within that range, remove the object directly from the pool, And by reference counting way to share, so that the system stores a large number of values, but also to some extent save memory and improve performance, this parameter value n settings need to modify the source code a line of macro definition redis_shared_integers, the value is 10000 by default, Can be modified according to their own needs, modified and re-compiled on it.

Persistence mechanism:

Timed snapshot mode (snapshot):

This persistence is actually a timer event within Redis, every fixed time to check whether the current data has changed the number of times and time to meet the configured persistence trigger condition, if satisfied then through the operating system fork to create a child process, This child process, by default, shares the same address space as the parent process, where it can traverse the entire memory for storage operations, while the main process can still provide the service, when there is write by the operating system according to the Memory page (page) Copy-on-write for the unit to ensure that the parent-child process does not affect each other.

The main disadvantage of this persistence is that the timed snapshot represents a memory image for a period of time, so the system reboot loses all data between the last snapshot and the restart.

Based on statement append mode (AOF):

The AOF approach is actually similar to MySQL's statement-based Binlog, where each command that causes Redis memory data to change is appended to a log file, which means that the log file is the persistent data for Redis.

The main disadvantage of the AOF approach is that appending the log file can lead to an excessive volume, and if it is aof when the system restarts the recovery data, it can be very slow to load data, and dozens of g of data may take a few hours to load, although this time is not due to slow disk file reads. Instead, all commands that are read are executed in memory. In addition, because each command has to write log, so the use of aof, Redis read and write performance will be reduced.

Consider saving the data to a different Redis instance, each with a memory size of around 2G, to avoid putting eggs in a basket, to reduce the impact of cache invalidation on the system, and to speed up data recovery, but it also brings some complexity to the system design.

Redis Persistence crash Issue:

People who have experience with Redis on-line operations will find that Redis has a lot of physical memory usage, but it has not exceeded the actual physical memory of the total capacity of the instability or even crash, some people think that is based on the snapshot persistence of the fork system calls resulting in double memory consumption, this view is inaccurate, Because the copy-on-write mechanism of the fork call is based on the unit of the operating system page, that is, only the dirty pages that are written will be copied, but generally your system does not write all pages in a short period of time, resulting in replication, then what causes Redis to crash?

The answer is that the persistence of Redis uses buffer Io, which means that Redis writes and reads to persisted files using the page Cache of physical memory, and most database systems use direct IO to bypass this layer of page Cache and maintain a cache of the data itself, and when a Redis persistent file is too large (especially a snapshot file) and read and write to it, the data in the disk file is loaded into physical memory as the operating system caches a layer of the file. And this cache of data and Redis in memory management data is actually repeatedly stored, although the kernel in the physical memory is tight when the page cache to do the culling work, but the kernel probably think that a piece of page cache more important, and let your process start swap, Your system will start to appear unstable or crash. Our experience is that when your Redis physical memory is using more than 3/5 of the total memory capacity, it starts to be more dangerous.

Summarize:

1, according to business needs to choose the appropriate data type, and for different application scenarios to set the corresponding compact storage parameters.

2. When the business scenario does not require data persistence, shutting down all persistence methods gives you the best performance and maximum memory usage.

3, if you need to use persistence, depending on whether you can tolerate the restart of the lost part of the data between the snapshot mode and the statement append mode, do not use virtual memory and Diskstore mode.

4. Do not let your redis machine physical memory use more than 3/5 of the total amount of actual memory.

The MaxMemory option in Redis.conf, which tells Redis how much physical memory is used to start rejecting subsequent write requests, which is good enough to protect your redis from the use of too much physical memory to cause swap, which can eventually seriously affect performance or even crash.

Vm-enabled No in redis.conf file