Redis Memory optimization

A recent system that uses Redis heavily, we store a large amount of user information in Redis, and the memory application is hundreds of G, and the volume is quite large. So we're constantly thinking about ways to optimize the use of redis memory and to share our optimization practices.

Use hash instead of <K,V> key value to store

Because it is the data that holds the user dimension, the user ID (UID) will often be the key, and a user will have more information, such as age, birthday, etc., the more easily thought of the storage structure will be hashed, a user's multiple information as a different field in the hash to store

Use Hash,list,zset's ziplist compression characteristics

Redis implements Ziplist compressed storage for hash,list,zset, which can be configured with a maximum of 512 elements and no more than 64bytes per element size to determine if the!ziplist compressed format is to be stored.

Note: Although this ziplist is enabled for configuration parameters, it is prudent to modify this configuration parameter because Ziplist is a contiguous array space, the lookup efficiency is not O (1), if the set element is more than 512 too many, may result in the search efficiency is reduced, but affect performance. So why is Redis using a default configuration like 512*64bytes? It is said that this size can be loaded into the CPU cache, so even if not O (1), the search efficiency is also very fast.

Prioritize using numeric types, which are more space-saving than string types

Within Redis, the string type, regardless of number type, is uniformly encapsulated with an object called Redisobject:

typedef struct REDISOBJECT {unsigned type:4;    unsigned encoding:4; unsigned lru:lru_bits;    /* LRU time (relative to Server.lruclock) */int refcount; void *ptr;} RobJ;

As can be seen, a simple "Hello World" in Redis is not directly 11 bytes is done, there are many additional properties, such as reference count (memory recycling) REFCOUNT,LRU cleanup and other information.

But if you use the Ziplist,redis mentioned above to cut the elements in the ziplist, so that the data is more compact, so for the number, do some special processing:

* |11000000| - 1 byte* integer encoded as int16_t  (2 bytes). * |11010000| - 1 byte* integer encoded as int32_t  (4 bytes). * |11100000| - 1 byte* integer encoded as int64_t  (8 bytes). * |11110000| - 1 byte* integer encoded as 24 bit signed   (3 bytes) .* |11111110| - 1 byte* integer encoded as 8  bit signed  (1 byte) .* |1111xxxx| -  (with xxxx between 0000  AND 1101)  immediate 4 bit integer.* Unsigned integer from 0  to 12. the encoded value is actually from* 1 to 13  because 0000 and 1111 can not be used, so 1 should be*  subtracted  From the encoded 4 bit value to obtain the right value. 

First use 1byte to represent different encode, for different sizes of the numbers, respectively, using the same memory space to store, such as 0-127 is 2 bytes, 128-32768 is 4 bytes and so on. So, in most cases, it's more memory-saving than string.

In addition, if you do not use ziplist storage, instead of using redisobject such a relatively large object storage?

If you can use numbers, try using a numeric type, and a number less than 10000 is best because:

#define Obj_shared_integers 10000

Redis takes into account the redisobject of this huge object that consumes too much memory, makes a pool of objects with numbers below 10000, and uses pointers (4/8bytes) to refer to the Redisobject in the pool, Instead of saving one copy of each.

Note: The above is for the previous version of Redis 3.2, because Redis 3.2 has made a lot of improvements to the Memory optimization section, and the specific improvement points are not yet known.

Finally, to insist on reading the classmate sent a very useful redis memory analysis tool: Redis-rdb-tools, combined with bgsave dump file, analysis of Redis data, you can see the underlying storage is what data structure, occupy the amount of space and other information.

Redis Memory optimization