Compact List of Redis implementations

Compression list

A compressed list (ziplist) is one of the underlying implementations of the list key and hash keys, when a list key contains only a few list items, and if each list item is an integer value, the shorter the string, then Redis uses the compression list to do the underlying implementation of the list key. For example, execute the following command to create an underlying implementation of a compressed list key

127.0.0.1:6379> RPUSH lst 1 3 5 10086 "hello" "world"
(integer) 6
127.0.0.1:6379> OBJECT ENCODING lst
"quicklist"

The quicklist structure in quicklist.c is interpreted as a doubly linked list of ziplists means a doubly linked list of ziplist, which contains the small integer values of 1, 3, 5, 10086, and the " A short string such as Hello "," World ".

In addition, when a hash key contains only a few keys and values, and the keys and values for each key pair are short strings, Redis uses a compression list to do the underlying implementation of the hash keys. Give me a chestnut. Execute the following command to create a hash key that is implemented by a compression list

127.0.0.1:6379> HMSET profile "name" "Jack" "age" 28 "job" "Programmer"
OK
127.0.0.1:6379> OBJECT ENCODING profile
"ziplist"

All keys and values contained in the hash key are small integers or short strings

Composition of the compression list

The compression list is a sequential data structure composed of a series of specially coded contiguous blocks of memory, developed by Redis to conserve memory. A compressed list can contain any number of nodes, each of which can hold a byte array or an integer value. Figure 1-1 shows the various components of the compression list, and table 1-1 records the type, length, and purpose of each component.

Figure 1-1 the various components of a compressed list

Table 1-1 Detailed description of the various components of the compression list

Property	Type	Length	Use
Zlbytes	uint32_t	4 bytes	Records the amount of memory bytes consumed by the entire compression list: When memory redistribution is performed on the compression list, or when calculating the location of the Zlend
Zltail	uint32_t	4 bytes	Records the number of bytes from the start address of the Compressed list footer node to the compressed list: This offset allows the program to determine the address of the footer node without traversing the entire compression list.
Zllen	uint16_t	2 bytes	Records the number of nodes that the compression list contains: When the value of this property is less than Uint16_max (65535), the value of this property is the number of nodes that the compression list contains; When this value equals Uint16_max, the actual number of nodes needs to traverse the entire compression list to calculate
Entryx	List node	Indefinite	The compression list contains the individual nodes, and the length of the node is determined by the contents of the node
Zlend	uint8_t	1 bytes	Special value 0xFF (decimal 255), used to mark the end of the compression list

Figure 1-2 shows an example of a compressed list:

• The value of the list Zlbytes property is 0x50 (decimal 80), which indicates that the total length of the compressed list is 80 bytes
• The value of the list Zltail property is 0x3c (decimal 60), which means that if there is a pointer p to the starting address of the compression list, the address of the footer node Entry3 can be calculated by using the pointer p plus the offset 60.
• The value of the list Zlend property is 0x3 (decimal 3), which indicates that the compression list contains three nodes

Figure 1-2 A compression list with three nodes

Figure 1-3 shows another example of a compression representation

Figure 1-3 A compression list with five nodes

• The value of the list Zlbytes property is 0xd2 (decimal 210), which indicates that the total length of the compressed list is 210 bytes
• The value of the list Zltail property is 0xb3 (decimal 179), which means that if there is a pointer p to the starting address of the compression list, the address of the footer node Entry5 can be calculated by using the pointer p plus the offset 179.
• The value of the list Zlend property is 0x5 (decimal 5), which indicates that the compression list contains five nodes

The composition of a compressed list node

Each compressed list node can hold a byte array or an integer value, where the byte array can be one of the following three lengths:

• byte array with length less than or equal to 63 (2^6-1) bytes
• byte array with length less than or equal to 16 383 (2^14-1) bytes
• byte array with length less than or equal to 4 294 967 295 (2^32-1) bytes

The integer value can be one of the following six lengths:

• 4 bits long, mind unsigned integers between 0 and 12
• 1-byte long signed integer
• 3-byte long signed integer
• int16_t type Integer
• int32_t type Integer
• int64_t type Integer

Each compressed list node consists of previous_entry_length, encoding, and content three parts, as shown in 1-4

Figure 1-4 Each component of a compressed list node

Previous_entry_length

The node Previous_entry_length property, in bytes, records the length of the previous node in the compressed list. The Previous_entry_length property can be 1 bytes or 5 bytes:

• If the length of the previous node is less than 254 bytes, the length of the Previous_entry_length property is 1 bytes: The length of the previous node is stored in this byte
• If the length of the previous node is greater than or equal to 254 bytes, the length of the Previous_entry_length property is 5 bytes: where the first byte of the property is set to 0xFE (decimal value 254), and the subsequent four bytes are used to hold the length of the previous node

Figure 1-5 shows a compressed list node with a one-byte long Previous_entry_length property with a value of 0x05, indicating that the previous node has a length of 5 bytes

Figure 1-5 The previous node of the current node has a length of 5 bytes

Figure 1-6 shows a compression node that contains a five-byte long Previous_entry_length property with a value of 0xfe00002766, where the highest byte 0xFE of the value indicates that this is a five-byte long Previous_entry_length property. The subsequent four-byte 00002766 (decimal 10086) is the actual length of the previous node. Because the Previous_entry_length property of the node records the length of the previous node, the program can calculate the starting address of the previous node based on the starting position of the current node, using the pointer operation.

For a chestnut, if we have a pointer to the starting address of the current node, C, then we simply subtract the value of the current node's Previous_entry_length property with Pointer C, we can draw a pointer to the start address of the previous node p,1-7 as shown in

Figure 1-7 Calculating the address of the previous node using pointer arithmetic

The table footer of a compressed list is implemented using this principle, as long as we have a pointer to the starting address of a node, then through this pointer and the node's Previous_entry_length property, the program can always forward a node backtracking, The table header node that eventually reaches the compression list.

Figure 1-8 shows a complete process of traversing from a footer node to a table head node:

• First, we have a pointer to the ENTRY4 starting address of the compressed list footer node P1 (pointers to footer nodes can be added to the Zltail property by pointers to the start address of the compression list)
• By subtracting the value of the Previous_entry_length property of the Entry4 node with P1, we get a pointer to the Entry3 start address of the Entry4 previous node P2
• By subtracting the value of the Previous_entry_length property of the Entry3 node with P2, we get a pointer to the Entry2 start address of the Entry3 previous node P3
• By subtracting the value of the Previous_entry_length property of the Entry2 node with P3, we get a pointer to the Entry1 start address of the Entry2 previous node p4,entry1 the header node of the compressed list
• Finally, we traverse the entire list from the footer node to the table head node

Figure 1-8 An example of a table header traversal from the end of the table

Encoding

The encoding property of a node records the type and length of data held by the content property of the node:

• One-byte, two-byte, or five-byte long, the highest bit of a value of 00, 01, or 10 is a byte-array encoding: This encoding means that the content property of the node holds the byte array, and the length of the array is removed by encoding the other bit records after the highest two bits
• One byte long, the highest value bit starting with 11 is the integer encoding: This encoding means that the content property of the node holds an integer value, the type and length of the integer value is removed by the encoding of the other bit records after the highest two bits

Table 1-2 records all available byte array encodings, and table 1-3 records all available integer encodings. The table "_" means empty, while the B, X and other variables represent the actual binary data, in order to facilitate reading, multiple bytes separated by a space

Table 1-2 byte array encoding

Coding	Encoding length	The Content property holds the value
00bbbbbb	1 bytes	byte array with length less than or equal to 63 bytes
01BBBBBB xxxxxxxx	2 bytes	byte array with length less than or equal to 16383 bytes
10______ aaaaaaaa bbbbbbbb CCCCCCCC dddddddd	5 bytes	byte array with length less than or equal to 4294967295

table 1-3 Integer encoding

Coding	Encoding length	The Content property holds the value
11000000	1 bytes	An integer of type int16_t
11010000	1 bytes	An integer of type int32_t
11100000	1 bytes	An integer of type int64_t
11110000	1 bytes	24-bit signed integer
11111110	1 bytes	8-bit signed integer
1111xxxx	1 bytes	The node that uses this encoding does not have a corresponding content property because the code itself has a value of XXXX four that has been saved between 0 and 12, so it does not need the content property

Content

The content property of a node is responsible for saving the node's value, which can be a byte array or an integer, and the type and length of the value is determined by the node's Encoding property

Figure 1-9 shows an example of a node that holds a byte array:

• The highest bit 00 of the encoding indicates that the node is holding a byte array
• The encoded post six bit 001011 records the length of the byte array 11
• The Content property holds the value of the node "Hello World"

Figure 1-9 The node that holds the byte array "Hello World"

Figure 1-10 shows an example of a node that holds an integer value:

• Encoding 11000000 means that the node holds an integer value of type int16_t
• The Content property holds the value of the node 10086

Figure 1-10 A node that holds an integer value of 10086

Chain Update

As I said earlier, the Previous_entry_length property of each node records the length of the previous node:

• If the length of the previous node is less than 254 bytes, then the Previous_entry_length property requires 1 bytes of space to hold the length value
• If the length of the previous node is greater than or equal to 254 bytes, then the Previous_entry_length property requires 5 bytes of space to hold the length value

Now, consider a situation in which there are multiple contiguous nodes between 250 bytes and 253 bytes in a compressed list, as shown in E1 to en,1-11

Figure 1-11 A compression list with E1 to EN

Because all nodes of the E1 to en are less than 254 bytes in length, the length of these nodes is only one byte long previous_entry_length attribute, in other words, the Previous_entry_length property of E1 to En is 1 bytes long. At this point, if we set a new node with a length greater than 254 bytes to the header node of the compressed list, then new will be the predecessor node of the E1, as shown in 1-12

Figure 1-12 Adding a new node to the compression list

Because E1 's Previous_entry_length property is only 1 bytes long, it has no way to save the new node's length, so the program will perform a spatial redistribution on the compressed list and will E1 node Previous_entry_ The length property expands from the original 1 byte length to 5 bytes long

Now, the trouble is, E1 the original length between 250 bytes to 253 bytes, after adding four bytes of space to the Previous_entry_length property, the length of E1 becomes between 254 bytes and 257 bytes. This length is not saved when using the 1-byte long previous_entry_length attribute. Therefore, in order for the Previous_entry_length property of E2 to record the length of the E1, the program needs to perform a spatial redistribution operation on the compressed list again and extend the Previous_entry_length attribute of the E2 node from the original 1 bytes to 5 bytes long

Just as extension E1 throws an extension to E2, extension E2 also throws an extension to the E3 ... In order for each node's Previous_entry_length property to conform to the requirements of the compression list for the node, the program needs to continuously perform a spatial redistribution operation on the compressed list until EN

Redis calls this continuous, multi-dimensional expansion operation in special cases called "Chain update", and figure 1-13 shows this process

Figure 1-13 Chain update process

In addition to adding nodes may cause cascading updates, deleting nodes may also cause cascading updates. Consider the compression list shown in Figure 1-14, if E1 to En is a node of size 250 bytes to 253 bytes, the big node is longer than or equal to 254 bytes (requires 5 bytes of previous_entry_length to save), While the length of the small node is less than 254 bytes (only 1 bytes of previous_entry_length are required to save), then when we remove the small node from the compression list, in order for E1 Previous_entry_ The Length property can record the size of the big node, and the program expands the space of the E1, which in turn causes the chain updates to follow.

Figure 1-14 The situation of another chain update

The worst-case complexity for cascading updates is O (n^2), because the cascading updates need to perform N-space redistribution operations on the compressed list at worst, while the worst time complexity for each spatial redistribution is O (N). It is important to note that, despite the high complexity of cascading updates, the probability that it really causes performance problems is very low:

• First of all, the compression list to have just a number of consecutive, length between 250 bytes to 253 bytes between the nodes, chain updates can be triggered, in practice, this can not be seen in many cases
• Second, even if there are cascading updates, the number of nodes that are updated will not have any effect on performance: for example, cascading updates to 35 nodes will never affect performance.

For these reasons, the average time complexity of commands such as Ziplistpush is only O (N), and in practice we can safely use these functions without worrying about cascading updates affecting the performance of the compression list

Compression List API

Table 1-4 lists all the APIs used to manipulate the compression list

Package 1-4 Compression List API

Function	Role	Complexity of Time
Ziplistnew (void)	Create a new compression list	O (1)
Ziplistpush (unsigned char *zl, unsigned char *s, unsigned int slen, int where)	Creates a new node with the given value and adds the new node to the header or footer of the compressed list	Average O (N), Worst O (n^2)
Ziplistinsert (unsigned char *zl, unsigned char *p, unsigned char *s, unsigned int slen)	Inserts a new node with the given value after the given node	Average O (N), Worst O (n^2)
Ziplistindex (unsigned char *zl, int index)	Returns a compressed list of nodes on a given index	O (N)
Ziplistfind (unsigned char *p, unsigned char *vstr, unsigned int vlen, unsigned int skip)	Finds and returns the node containing the given value in the compressed list	Because the value of a node can be a byte array, it is O (N) to check whether the node value is the same as the given value, and the complexity of finding the entire list is O (n^2)
Ziplistnext (unsigned char *zl, unsigned char *p)	Returns the next node of a given node	O (1)
Ziplistprev (unsigned char *zl, unsigned char *p)	Returns the previous node of a given node	O (1)
Ziplistget (unsigned char *p, unsigned char **sval, unsigned int *slen, long long *lval)	Gets the value saved by the given node	O (1)
Ziplistdelete (unsigned char *zl, unsigned char **p)	Remove a given node from the compressed list	Average O (N), Worst O (n^2)
Ziplistdeleterange (unsigned char *zl, unsigned int index, unsigned int num)	Deletes a contiguous number of nodes on a given index for a compressed list	Average O (N), Worst O (n^2)
Ziplistbloblen (unsigned char *zl)	Returns the number of memory bytes currently occupied by the compression list	O (1)
Ziplistlen (unsigned char *zl)	Returns the number of nodes currently contained in the compression list	Number of nodes is less than 65535 O (1), greater than 65535 O (N)

Compact List of Redis implementations