Make memory management based on Redis

1 Redis storage mechanism:

data types for Redis storage include,String,Hash,List,Set , Sorted Set, which internally uses a Redisobject object to represent all keys and value, The basic structure of this object is shown below:

typedef struct REDISOBJECT {

unsigned type,//4 bytes, data type

unsigned encoding,//4 Bytes, encoding method

unsigned LRU,// bytes, permutation algorithm

int RefCount,// object reference count

void *ptr// data-specific storage pointing to

} robj;

Type represents a value object specifically what data type,encoding is different data type in redis Internal storage, so that the main purpose is to provide a unified management interface for different data types of Redis.

String is the most common type, is the most common key/value hash storage, read and write time complexity is O (1).

Hash store will be in basic kv hash hash hash structure, can be used to direct the value of the id+key value O (1)

List Stores a doubly linked list, which can support reverse lookup and traversal, the time complexity of inserting a data into a query O (n) If it was added or popped last o (1)

Set stored is an auto-de-weight List set stored is a value of null map set supports the existence of direct judgment values, set The time complexity of reading and writing is o (1)

Sorted Set is stored in the order of the user's specified score Set,theSorted set store uses a hop table, and a hash, the Skip table stores all the data,hash Stores the relationship between data and fractions. Sorted Set reads and writes are mainly the complexity of querying data or locations from a jump table, which is O (log (n)).

8.2 Redis Memory management mechanism:

Memory Control:

      Redis implements memory management by controlling memory caps and recycling policies, using maxmemory parameters limit the maximum available memory, such as our nfvo 512mb represents our largest cannot store more than 512mb data.

for outdated data,Redis has two ways to free memory, one is lazy delete, the next time you query a key to check the time-out, if it has timed out, delete the data and return NULL, the other is a scheduled task, the default per ten seconds (can be configured with the Hz parameter, we use the default configuration of ten), check the time-out period, the timeout is deleted.

There are several ways to handle overflow data, which can be configured through maxmemory-policy .

1, noeviction, do not delete, reject the newly inserted message.

2, Volatile-lru, according to the LRU(least recently used) permutation algorithm , for the configured time-out data to clean up, this is our current configuration of the cleanup policy, The frequent occurrence of data loss during the show is due to the existence of this strategy, so be careful when using it.

3, Allkeys-lru, according to the LRU(least recently used) permutation algorithm, for all data to clean up until the memory to meet the needs.

4, Allkeys-random, randomly delete all keysuntil memory needs are met.

5, Volatile-random, randomly deletes the key configured with the timeout period until the memory needs are met.

6, Volatile-ttl, according to the key object's time-to-live property, delete the data that will expire recently.

when a client executes a new command that causes the database to add data (such as set key value),Redis checks memory usage if memory usage exceeds maxmemory , some keys are removed according to the substitution policy to prepare the store for the new data.

Redis also saves memory by using different processing of size data, such as Several typesof Redis mentioned above,list,set, Map, when the data volume is small , Redis uses linear compact storage to save space, and the amount of data is configurable:

Entries means that when value is not more than the number of members within the MAP is stored in a linear compact format, the default is value means that a Linear Compact storage is used to save space when value does not exceed the number of bytes per member within the MAP. , this value cannot be infinitely enlarged, because linear storage means linear lookups, less data is more efficient, and when the data is large, it can reduce query efficiency.

8.3 Redis persistence mechanism:

Redis supports a total of four persistence modes, namely , timed snapshot mode (snapshot), the way the file is appended based on the statement (aof), and virtual memory ( vm),diskstore mode. The main use is the first two,the VM mode has been discarded, and diskstore is only in the experimental phase,that is,Redis Currently used as a memory database, the amount of data is not allowed to be large.

timed Snapshot modeis actually inRedisInternal one Timer event, every fixed time to check whether the current data changes the number and time to meet the configured persistence trigger condition, if satisfied then through the operating systemForkCalled to create a child process that, by default, shares the same address space with the parent process, can traverse the entire memory through the child process to perform the storage operation, while the main process can still provide the service, when there is write by the operating system according to the memory page (page) for the unit.Copy-on-writeensure that parent-child processes do not affect each other. The main disadvantage of this persistence is that the timed snapshot represents a memory image for a period of time, so the system reboot loses all data between the last snapshot and the restart. Timed snapshot mode is based onredis.confconfigured in theSavetime interval to check whether the current data change times and times are satisfied with the configuration, and if satisfied, from the parent processFork (Copy-on-writemechanism)a child process that iterates through the memory to convert theRdbfiles, inredis.confSee you in the following:

Represents 900 respectivelyin seconds1aKeyChange, -in secondsTenaKey, -in seconds10000aKeytriggered, due to ourNFVOand theVNFMbecause it cleans up every time .Redis, the persistence mechanism is not used, so the configuration is dropped.

AoFWayeach one will makeRediscommands that change memory data are appended to aLogfile, which means thisLogfile isRedisof persistent data,aofThe main disadvantage of the way is to appendLogthe file can cause excessive volume, and if the system restarts the recovery dataaof, loading the data is very slow because all the commands that are read are executed once in memory. Plus, because every command is written ,Log, so useaofthe way thatRedisRead and write performance will also decrease. aofThere are three modes of invocationFsyncwrite to disk,Fsyncfunction can synchronize all the modified file data in memory to the storage device,1, always write, that is, each time a change is calledFsyncwrite;2, calls per secondFsyncWrite once;3, do not actively invokeFsyncsynchronizing data to disk, but relying onOsthe timingFsyncwrite, generally for -seconds, not recommended. aofWay inredis.confinside configuration See below:

AppendOnly represents whether to open AOF, we do not open,appendfilename is the file name, Appendfsync is a write mode.

Redis-based memory management