Redis Data Persistence

Redisobject

First, Redis internally uses a Redisobject object to represent all key and value,redisobject most important information as shown in the figure above: type represents what data type a value object is, Encoding is how different data types are stored inside the Redis, such as: Type=string represents a normal string for value, then the corresponding encoding can be raw or int, If it is an int, the actual redis interior is stored and represented by a numeric class, assuming that the string itself can be represented numerically, such as a string such as "123" "456".
five types of data and how to implement it internally

string Implementation method: String in the Redis internal storage By default is a string, is referenced by the Redisobject, when encountering INCR,DECR and other operations will be converted to a numerical type for calculation,

At this point the encoding field of Redisobject is int. Hash Redis provides an interface (Hgetall) that can fetch all of the property data directly, but if the internal map has a large number of members, then it involves traversing the entire internal map operation, due to the **redis single-threaded model * *, this traversal operation may be time-consuming,

The other client requests are not responding at all, which requires extra attention. Implementation: The above has been said that the Redis hash corresponds to value inside the actual is a hashmap, actually there will be 2 different implementations, this hash of the members of the relatively small redis in order to save memory will be similar to a one-dimensional array to compact storage,

Instead of a true hashmap structure, the corresponding value Redisobject encoding is Zipmap, and when the number of members increases, it automatically turns into a real hashmap, at which time encoding is HT.

List implementation: the implementation of the Redis list is a * * doubly linked list * *, which can support reverse lookup and traversal, more convenient operation, but with some additional memory overhead, many implementations within Redis, including sending buffer queues, are also used in this data structure.

Set Implementation method: The internal implementation of the set is a value is always null **hashmap**, is actually calculated by calculating the hash of the way to fast weight, which is also set to provide a judge whether a member is within the set of reasons. Sorted Set Implementation method: Redis Sorted Set of * * Internal use HashMap and jump Table (skiplist) * * To ensure the storage and ordering of data, HashMap put the member to score mapping, and the jumping table is stored all the members, Sorting is based on the score of HashMap, using the structure of the jumping table to obtain a relatively high efficiency, and the implementation of the simple 

the last thing to say is that redis internal implementations do not optimize memory allocations to some extent, but in most cases this will not be a performance bottleneck for Redis, but if most of the data stored inside Redis is numeric, The Redis interior employs a shared integer to eliminate the overhead of allocating memory, which is to assign a number of numeric objects from 1~n to a pool when the system starts, and if the stored data happens to be data within that range, remove the object directly from the pool, And by reference counting way to share, so that the system stores a large number of values, but also to some extent save memory and improve performance, this parameter value n settings need to modify the source code a line of macro definition redis_shared_integers, the value is 10000 by default, Can be modified according to their own needs, modified and re-compiled on it. The persistence mechanism of Redis

Since Redis supports very rich types of memory data structures, how to persist these complex memory organizations to disk is a challenge, so there are more differences in how redis is persisted than traditional databases, and Redis supports four persistence modes, namely:

How the timed snapshot mode (snapshot)
is based on the statement append file (AOF)
virtual memory (VM)
Diskstore mode

In the design thinking, the first two are based on all the data in memory, that is, the small amount of data to provide disk landing function, the latter two ways is the author in the attempt to store data over physical memory, that is, the large amount of data storage , as of this article, the last two persistent mode is still in the experimental phase, And the VM way basically has been abandoned by the author, so the actual can be used in the production environment only the first two, in other words, Redis is currently only as a small amount of data storage (all data can be loaded in memory), massive data storage is not the domain that Redis excels at. Here are some of the following methods of persistence: timed Snapshot mode (snapshot):

This persistence is actually a timer event within Redis, every fixed time to check whether the current data has changed the number of times and time to meet the configured persistence trigger condition, if satisfied then through the operating system fork to create a child process, This child process, by default, shares the same address space as the parent process, where it can traverse the entire memory for storage operations, while the main process can still provide the service, when there is write by the operating system according to the Memory page (page) Copy-on-write for the unit to ensure that the parent-child process does not affect each other .

The main disadvantage of this persistence is that the timed snapshot represents a memory image for a period of time, so the system reboot loses all data between the last snapshot and the restart. based on statement append mode (AOF):

The aof approach is actually similar to MySQL's statement-based Binlog, where each command that causes Redis memory data to change is appended to a log file, which means that the log file is the persistent data for Redis .

The main disadvantage of the AOF approach is that appending the log file can lead to an excessive volume, and if it is aof when the system restarts the recovery data, it can be very slow to load data, and dozens of g of data may take a few hours to load, although this time is not due to slow disk file reads. Instead, all commands that are read are executed in memory. In addition, because each command has to write log, so the use of aof, Redis read and write performance will be reduced . virtual memory mode:

The virtual memory mode is a redis for user space data exchange in a policy, this way in the implementation of the effect is poor, the main problem is the code complex, restart slow, replication slow, etc., has been abandoned by the author. Diskstore Way:

Diskstore Way is the author abandoned the virtual memory mode after the choice of a new way of implementation, that is, the traditional way of B-tree, is still in the experimental stage, the follow-up is available we can wait and see. redis Persistent Disk IO mode and the problems it brings

People who have experience with Redis on-line operations will find that Redis has a lot of physical memory usage, but it has not exceeded the actual physical memory of the total capacity of the instability or even crash, some people think that is based on the snapshot persistence of the fork system calls resulting in double memory consumption, this view is inaccurate, Because the copy-on-write mechanism of the fork call is based on the unit of the operating system page, that is, only the dirty pages that are written will be copied, but generally your system does not write all pages in a short period of time, resulting in replication, then what causes Redis to crash.

The answer is that the persistence of Redis uses buffer Io, which means that Redis writes and reads to persisted files using the page Cache of physical memory, and most database systems use direct IO to bypass this layer of page Cache and maintain a cache of the data itself, and when a Redis persistent file is too large (especially a snapshot file) and read and write to it, the data in the disk file is loaded into physical memory as the operating system caches a layer of the file. And this cache of data and Redis in memory management data is actually repeatedly stored, although the kernel in the physical memory is tight when the page cache to do the culling work, but the kernel probably think that a piece of page cache more important, and let your process start swap, Your system will start to appear unstable or crash . Our experience is that when your Redis physical memory is using more than 3/5 of the total memory capacity, it starts to be more dangerous.