Redis memory usage optimization and storage

Before describing these types of data, let's look at a diagram of how these different data types are described in Redis internal memory management:

First, Redis internally uses a Redisobject object to represent all key and value,redisobject information as shown in: type represents what data type a value object is, Encoding is how different data types are stored inside the Redis, such as: Type=string represents a normal string for value, then the corresponding encoding can be raw or int, If it is an int, the actual redis interior is stored and represented by a numeric class, assuming that the string itself can be represented numerically, such as a string such as "123" "456".

Here you need to specify the VM field, only the virtual memory feature of Redis is turned on, this field will actually allocate memory, which is turned off by default, which is described later in this function. We can find that Redis uses Redisobject to indicate that all key/value data is a waste of memory, and of course, the cost of memory management is mainly to provide a unified management interface for different data types of Redis. The actual author also offers several ways to help us save memory as much as possible, which we'll discuss in detail later.

Let's start with the analysis of the use of these five types of data and how to implement them internally:

• String

  Common commands:

  Set,get,decr,incr,mget and so on.

  Application Scenarios:

  String is the most commonly used type of data, and ordinary key/value storage can be categorized as such, which is not explained here.

  Implementation method:

  String in the Redis internal storage By default is a string, referenced by Redisobject, when encountered INCR,DECR and other operations will be converted to a numeric type for calculation, at this time Redisobject encoding field is an int.

• Hash

  Common commands:

  Hget,hset,hgetall and so on.

  Application Scenarios:

  Let's simply cite an example to describe the application scenario for a hash, such as storing a user information object data that contains the following information:

  The user ID is the key to find, the stored value user object contains the name, age, birthday and other information, if the ordinary key/value structure to store, mainly has the following 2 kinds of storage methods:

  The disadvantage of using the user ID as a lookup key to encapsulate other information as a serialized object is to increase the cost of serialization/deserialization and to retrieve the entire object when one of the information needs to be modified, and the modification operation requires concurrency protection. Introduce complex problems such as CAs.

  The second method is how many members of this user information object will be saved into the number of key-value, with the user id+ the name of the corresponding property as a unique identifier to obtain the value of the corresponding property, although the cost of serialization and concurrency is omitted, but the user ID is repeated storage, if there is a large number of such data, The memory waste is still very considerable.

  So the hash provided by Redis is a good solution to this problem, and the Redis hash is actually the internal stored value as a hashmap, and provides a direct access to the map member's interface, such as:

  That is, the key is still the user ID, value is a map, the map key is a member of the property name, value is the property value, so that the data can be modified and accessed directly through its internal map key (Redis called internal map key field), This means that the corresponding attribute data can be manipulated by key (user ID) + field (attribute tag), without the need to store the data repeatedly and without the problem of serialization and concurrency modification control. A good solution to the problem.

  It is also important to note that Redis provides an interface (Hgetall) that can fetch all of the property data directly, but if the internal map has a large number of members, it involves traversing the entire internal map, which can be time-consuming due to the Redis single-threaded model. The other client requests are not responding at all, which requires extra attention.

  Implementation method:

  The above has been said that the Redis hash corresponds to value inside the actual is a hashmap, actually there will be 2 different implementations, this hash of the members of the relatively small redis in order to save memory will be similar to a one-dimensional array to compact storage, without the use of a real HASHMAP structure , the encoding of the corresponding value Redisobject is Zipmap, and when the number of members increases, it automatically turns into a true hashmap, at which time encoding is HT.

• List

  Common commands:

  Lpush,rpush,lpop,rpop,lrange and so on.

  Application Scenarios:

  Redis list has many applications and is one of the most important data structures of redis, such as Twitter's watchlist, fan list, etc., which can be implemented using the REDIS list structure, which is better understood and not repeated here.

  Implementation method:

  The implementation of Redis list is a doubly linked list, which can support reverse lookup and traversal, but it is more convenient to operate, but it brings some additional memory overhead, and many implementations within Redis, including sending buffer queues, are also used in this data structure.

• Set

  Common commands:

  Sadd,spop,smembers,sunion and so on.

  Application Scenarios:

  The functionality provided by Redis set externally is a list-like feature, except that set is automatically weight-saving, and set is a good choice when you need to store a list of data and you don't want duplicate data. and set provides an important interface to determine whether a member is within a set set, which is not available in list.

  Implementation method:

  The internal implementation of set is a value that is always null hashmap, which is actually calculated by hashing the way to fast weight, which is also set to provide a judge whether a member is within the cause of the collection.

• Sorted Set

  Common commands:

  Zadd,zrange,zrem,zcard, etc.

  Usage scenarios:

  The usage scenario for Redis sorted set is similar to set, except that the set is not automatically ordered, and the sorted set can be ordered by the user with an additional priority (score) parameter, and is inserted in an orderly, automatic sort. When you need an ordered and non-repeating collection list, you can choose sorted set data structures, such as the public Timeline of Twitter, which can be stored as score in the publication time, which is automatically sorted by time.

  Implementation method:

  Redis sorted set internal use HashMap and jump Table (skiplist) to ensure the storage and ordering of data, HashMap in the member to score mapping, and the jumping table is all the members, sorted by HashMap in the score , the use of the structure of the jumping table can obtain a relatively high efficiency of finding, and it is relatively simple to implement.

Redis memory usage optimization and storage