The object of the Redis implementation (iii)

Collection Object

The encoding of a collection object can be either Intset or Hashtable,intset-encoded collection objects using an integer collection as the underlying implementation, and all the elements contained in the collection object are stored in the set of integers. For a chestnut, the following code creates a Intset encoded collection object as shown in Figure 1-12:

127.0.0.1:6379> Sadd numbers 1 3 5 (integer) 3127.0.0.1:6379> OBJECT ENCODING Numbers "Intset"

　　　　

Figure 1-12 Inset encoded numbers collection Object

On the other hand, the Hashtable encoded collection object uses a dictionary as the underlying implementation, each key of the dictionary is a string object, each string object contains a collection element, and the value of the dictionary is all set to NULL, the following example, A Hashtable encoded collection object, shown in 1-13, is created:

127.0.0.1:6379> sadd Fruits "apple" "Banana" "Cherry" (integer) 3127.0.0.1:6379> OBJECT ENCODING fruits "Hashtable"

　　

Figure 1-13 Hashtable encoded Fruits Collection Object

Encoded conversions

The object uses Intset encoding when the collection object can meet the following two conditions:

• All elements saved by the collection object are integer values
• Collection object holds fewer than 512 elements

The collection object that cannot satisfy the above two condition pairs needs to use Hashtable encoding, note that the first condition cannot be modified, but the upper value of the second condition can be modified, see the description of the Set-max-intset-entries option in the configuration file

For a collection object that uses Intset encoding, when any one of the two conditions required to use the Intset encoding cannot be satisfied, the encoding conversion of the object is performed, and all the elements that were stored in the integer collection are transferred and saved in the dictionary. and the encoding of the object will change from Intset to Hashtable

For a chestnut, the following code creates a collection object that contains only an integer element, originally encoded as intset, but as soon as we add a string element, the encoding transfer operation of the collection object is executed

127.0.0.1:6379> Sadd numbers 1 3 5 (integer) 3127.0.0.1:6379> OBJECT ENCODING Numbers "Intset" 127.0.0.1:6379> Sadd numbers "seven" (integer) 1127.0.0.1:6379> OBJECT ENCODING Numbers "Hashtable"

　　

In addition, if we create a collection object that contains 512 integer elements, the object's encoding should be intset. However, as soon as we add an integer element to the collection so that the element of the collection becomes 513, the object's encoding conversion operation is executed:

127.0.0.1:6379> EVAL "for I=1, + redis.call (' Sadd ', keys[1], i) end" 1 integers (nil) 127.0.0.1:6379> SCard Integ ERs (integer) 512127.0.0.1:6379> OBJECT ENCODING integers "intset" 127.0.0.1:6379> sadd integers 10086 (integer) 1127.0.0.1:6379> scard integers (integer) 513127.0.0.1:6379> OBJECT ENCODING integers "hashtable"

　　

Implementation of the collection command

Because the value of the collection key is a collection object, all commands for the collection key are manipulated against the collection object, and table 1-10 lists the commands for some of the collection keys and how they are implemented under different encoded collection objects

table 8-10 How to implement a collection command

Command	Implementation method of Intset coding	Implementation method of Hashtable coding
Sadd	Call the Intsetadd function to add all new elements to the set of integers	Call Dictadd, with new element as key, NULL as value, add key value pair to dictionary
SCard	Call the Intsetlen function to return the number of elements contained in an integer collection, which is the number of elements contained in the collection object	Call the Dictsize function to return the number of key-value pairs that the dictionary contains, which is the number of elements contained in the collection object
Sismember	Call the Intsetfind function to find the given element in the integer collection, if a description element is found to exist in the collection, the element does not exist in the collection	Call the Dictfind function to find the given element in the key of the dictionary, if a description element is found to exist in the collection, the element does not exist in the collection
Smembers	Iterate through the whole set of integers, using the Intsetget function to return the collection elements	Iterate through the dictionary, using the Dictgetkey function to return the key of the dictionary as a collection element
Srandmember	Call the Intsetrandom function to return an element randomly from the set of integers	Call the Dictgetrandomkey function to return a dictionary key randomly from the dictionary
SPOP	Call the Intsetrandom function to randomly remove an element from an integer collection, and after returning the random element to the client, call the Intsetremove function to remove the random element from the collection of integers	Call the Dictgetrandomkey function, a dictionary key is randomly fetched from the dictionary, and after returning the value of the random dictionary key to the client, call the Dictdelete function to remove the key value pair from the dictionary for the Random dictionary key
Srem	Call the Intsetremove function to remove all the given elements from the integer collection	Call the Dictdelete function to remove all key-value pairs from the dictionary for the given element

Ordered collection objects

The encoding of an ordered collection can be ziplist or skiplist,ziplist encoded compression lists, each set element is saved with two compressed list nodes that are close together, and the first node holds the members of the element (member). The second element holds the element's score (score). The collection elements in the compressed list are sorted from small to large, with smaller points placed in the direction near the head, while elements with larger scores are placed near the end of the table.

Give me a chestnut. If we execute the following Zadd command, the server creates an ordered collection object as the value of the price key:

127.0.0.1:6379> Zadd Price 8.5 Apple 5.0 Banana 6.0 Cherry (integer) 3127.0.0.1:6379> OBJECT ENCODING price "Ziplist"

　　

Price This value is shown in Object 1-14, while the object is using a compression list of 1-15

Figure 1-14 An ordered set of Ziplist encoded objects

Figure 1-15 ordered collection elements are sorted from small to large in the compression list

The Skiplist encoded ordered collection object uses the ZSET structure as the underlying implementation, and a ZSET structure contains both a dictionary and a skip table

Redis.h

typedef struct ZSET {    dict *dict;    Zskiplist *ZSL;} zset;typedef struct Zskiplistnode {    robj *obj;    Double score;    struct Zskiplistnode *backward;    struct Zskiplistlevel {        struct zskiplistnode *forward;        unsigned int span;    } Level[];} zskiplistnode;typedef struct Zskiplist {    struct zskiplistnode *header, *tail;    unsigned long length;    int level;} Zskiplist;

　　

The ZSL jumping table in the ZSET structure saves all collection elements from small to large, and each skip table node holds a collection element: The obj attribute of the Skip table node holds the members of the element, and the score attribute of the Jump table node preserves the element's score. Through this jumping table, the program can do the scope of the ordered set of operations, such as Zrank, Zrange and other commands are based on the Jumping table API to achieve

In addition, the Dict dictionary in the ZSET structure creates a mapping from member to score for an ordered set, and each key-value pair in the dictionary holds a collection element: The dictionary key holds the member of the element, and the value of the dictionary preserves the element's score. Through this dictionary, the program can find the score of a given member in the time Complexity of O (1), the Zscore command is implemented according to this feature, and many other ordered set commands are used within the implementation of this feature

The members of each element in an ordered collection are a string object, and the score for each element is a double type of floating-point number. It is worth mentioning that although the Zset structure uses both a skip table and a dictionary to hold an ordered set element, both data structures share the members and scores of the same elements through pointers, so using a skip table and a dictionary to save the collection element does not result in any duplicate members or scores, nor does it waste additional memory

Why do ordered collections need to be implemented using both jump tables and dictionaries? In theory, an ordered set can be implemented by using either a dictionary or a data structure of a jumping table, but whether using a dictionary or a skip table, the performance is reduced by using both a dictionary and a skip table. For example, if we are only using a dictionary to implement an ordered set, then although we can find the corresponding score of the member in the time Complexity of O (1), however, because the dictionary saves the collection elements in an unordered manner, each time you perform a range-based operation such as Zrank, Zrange, etc. The program needs to sort all the elements of the dictionary, which requires at least O (n logn) of time complexity and an additional O (n) memory space (because an array is created to hold the sorted elements)

On the other hand, if we only use the Jumping table to implement an ordered set, then all the advantages of the jump table in the scope operation will be preserved, but because there is no dictionary, the time complexity of the operation to find the score based on the member is increased from O (1) to O (Logn). For these reasons, Redis chooses to use both a dictionary and a jump table to implement an ordered set of data structures in order for the lookup and scope operations of an ordered collection to be executed as quickly as possible.

For a chestnut, if the previous price key creates an ordered collection object that is not ziplist encoded, but an ordered collection object that is skiplist encoded, the ordered collection object will look like the one shown in Figure 1-16, and the zset structure used by the object will look like Figure 8-17.

Figure 1-16 An ordered set of Skiplist encoded objects

Figure 1-17 ordered collection elements are stored in dictionaries and jump tables

Encoded conversions

An object uses Ziplist encoding when an ordered collection object can meet both of the following conditions:

• The number of elements saved by an ordered collection is less than 128
• The length of all elements saved by an ordered collection is less than 64 bytes

An ordered collection object that cannot meet the above two conditions will use the Skiplist encoding

# object contains 128 elements 127.0.0.1:6379> EVAL "for I=1, X-do Redis.call (' Zadd ', keys[1], I, i) end" 1 numbers (nil) 127.0.0.1:637 9> zcard Numbers (integer) 128127.0.0.1:6379> OBJECT ENCODING Numbers "ziplist" # Add a new element 127.0.0.1:6379> Zadd Numbers 3.14 Pi (integer) # object contains the number of elements to 129 127.0.0.1:6379> Zcard numbers (integer) 129# encoding changed 127.0.0.1:6379> objec T ENCODING Numbers "Skiplist"

　　

The following code shows the case of an ordered collection object that throws an encoding conversion because the member of the element is too long:

# Add a member to an ordered collection only three-byte long elements 127.0.0.1:6379> zadd blah 1.0 www (integer) 1127.0.0.1:6379> OBJECT ENCODING blah "ziplist" # Add an element with a 66-byte member to an ordered collection 127.0.0.1:6379> zadd blah 2.0 Oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo (integer) # code has changed 127.0.0.1:6379> OBJECT ENCODING blah "skiplist"

　　

Implementation of ordered SET command

Because the value of an ordered collection key is a hash value, all commands for an ordered collection key are built on a hash object, and table 1-11 lists some of the ordered set key commands, and how they are implemented under different encoded hash objects

table 8-11 Implementation method of ordered set command

Command	Implementation method of Ziplist coding	Implementation method of Zset coding
Zadd	Call the Ziplistinsert function to insert a member and a score as two nodes into the compression list	Call the Zslinsert function first, add the new element to the jump table, and then call the Dictadd function to associate the new element to the dictionary
Zcard	Call the Ziplistlen function to get the number of compressed list containing nodes, dividing this number by 2 to derive the number of collection elements	Accesses the length property of a skip table data structure, directly returning the number of collection elements
Zcount	Iterate through the compression list, counting the number of nodes within a given range	Iterate over a jump table, counting the number of nodes in a given range
Zrange	Iterates through the compression list from the table header to the end of the table, returning all elements in the given index range	Iterates through the jump table from the table header to the end of the table, returning all elements in the given index range
Zrevrange	Iterates through the compressed list from the footer to the table header, returning all elements in the given index range	Iterates through the jump table from the footer to the header, returning all elements in the given index range
Zrank	Traverse the compression list from the table header to the end of the table, find the given member, record the number of nodes along the way, and when a given member is found, the number of passing nodes is the rank of the element corresponding to that member	Traverse the jump table from the table header to the end of the table, find the given member, record the number of nodes along the way, and when a given member is found, the number of passing nodes is the rank of the element corresponding to that member
Zrevrank	Traverse the compression list from the end of the table to the table header, find the given member, record the number of nodes along the way, and when a given member is found, the number of passing nodes is the rank of the element corresponding to that member	Traverse the jump table from the end of the table to the table header, find the given member, record the number of nodes along the way, and when a given member is found, the number of passing nodes is the rank of the element corresponding to that member
Zrem	Traverse the compression list, delete all nodes containing the given member, and the node of the node next to the deleted member	Iterate through the Skip table and delete all the jump table nodes that contain the given member. and disassociate the members and scores of the deleted elements in the dictionary
Zscore	Traverse the compression list, find the node that contains the given member, and then take out the element score saved by the score node next to the member node	Remove a given member's score directly from the dictionary

The object of the Redis implementation (iii)