Redis Topic 1: Data Structure and redis topic Data Structure

Redis Topic 1: Data Structure and redis topic Data Structure
Simple Dynamic string

• Redis does not directly use the traditional string representation in the C language, but builds a dynamic string SDS. When redis requires more than a string literal, however, when a string value can be used by the show ide, redis uses sds to represent the string value, for example, in the redis database, key-value pairs that contain string values are implemented by SDS at the underlying layer.

Redis> set name "bugall" ok1. the key of a key-value pair is a string object, the underlying Implementation of the object is the value of the SDS2. key-value pair that stores the string "name" and also a String object. The underlying Implementation of the object is an SDS that saves the string "bugall"

• In addition to storing string values in the database, SDS is also used as a buffer, AOF buffer in the AOF module, and input buffer in the client status.
  

Struct sdshdr {// record the number of bytes used in the buf array // equal to the length of the string stored by SDS int len; // record the number of unused bytes in the buf array int free; // byte array, used to save the string char buf [];} 1. the value of the free attribute is 0, indicating that this SDS is not allocated any unused space. 2. the value of the len attribute is 5, indicating that this SDS stores a five-byte long string 3. the buf attribute is an array of the char type. The first five bytes of the array contain five characters: 'R', 'E', 'D', 'I', and 'S, the last byte stores the null character '\ 0'. Note: Saving the 1-byte space of Confucius' is not included in the len attribute of SDS, additionally, an extra byte of space is allocated for null characters.

• Unlike the c string, the space allocation policy of SDS completely eliminates the possibility of buffer overflow. When the SDS api needs to modify the SDS, the API first checks whether the SDS space meets the modification requirements. If not, the API automatically expands the SDS space to the size required for modification, then, the actual modification is performed. Therefore, you do not need to manually modify the space size of SDS without any buffer overflow.

• Space allocation policy:
  To avoid this defect of the C string, SDS removes the association between the string length and the underlying array length without using space. In SDS, the length of the buf array is not necessarily equal to the number of characters plus one, the array can contain unused bytes, and the number of these bytes is recorded by the fress attribute of SDS. Through space not used, SDS implements two optimization policies: pre-allocation of space and release of inert space.

• Space pre-allocation:
  Space pre-allocation is used to optimize the string growth operation of SDS: When the sds api modifies an SDS and needs to expand the space of the SDS, the program not only allocates the space required for modification to SDS, but also allocates additional unused space for SDS. Through the space pre-allocation policy, redis can reduce the number of times the memory is re-allocated for consecutive string growth operations.

• Inert space release
  The release of the inert space is used to optimize the SDS string shortening operation: When the sds api needs to shorten characters, the program does not immediately use memory reallocation to recycle the shortened bytes, instead, record the number of these bytes (free) and wait for future use.

• SDS is still binary secure and supports C string functions.

Linked List

• The linked list provides efficient node shuffling capabilities and sequential node access methods. You can also flexibly adjust the length of the Linked List by adding or deleting nodes. Because Redis uses a C language that does not have a built-in data structure, Redis has built its own linked list implementation. The linked list is widely used in Redis. For example, one of the underlying implementations of the list key is the linked list. When a list key contains a large number of elements, or, when all the elements in the list are long strings, Redis uses the linked list as the underlying implementation of the list key.

• The underlying implementation of the integers list key is a linked list. each node in the linked list stores an integer. In addition to the linked list key, functions such as publishing and subscription, slow query, and monitor also use the linked list. The Redis server itself uses the linked list to store the status information of multiple clients, and use the linked list to build the output buffer of the client.

Typeof struct list {// the header uses the listNode * head // The number of nodes contained in the listNode * tail // linked list at the end of the table. The unsigned long len // node value copying function void *( * dup) (void * ptr); // void (* free) (void * ptr) of the node value release function; // void (* match) (void * ptr, void * key)} list

Dictionary

• A dictionary, also known as a symbol table, is an abstract data structure used to store key-value pairs. Redis databases use dictionaries as the underlying layer. In addition to representing the database, the dictionary is also one of the underlying implementations of the hash key. When a hash key contains many key-value pairs, or when the elements in the key-value pair are long strings. Redis uses the dictionary as the underlying implementation of the hash key. For example:

A ={} a {users: {name: 'bucall', name: 'ifang '} among them, 'users' can be considered as a stable hash key, the key-value pair is the content corresponding to 'users.

• The Redis dictionary uses a hash table as the underlying implementation. A hash table can contain multiple hash table nodes, and each hash table node stores a key-value pair in the dictionary.
  

Typedef struct dictht {// hash table array dictEntry ** table; // hash table size: unsigned long size; // hash table size mask, used to calculate the index value // always equal to size-1 unsigned long sizemaske; // number of existing nodes in the hash table unsigned long used;} dictht; hash table node: typedef struct dictEntry {// key void * key // value union {void * val; uint64_tu64; int64_ts64;} v; struct dictEntry * next;} dictEntry; the next attribute is a pointer to another hash table node. This pointer can be used to connect multiple key-value pairs with the same hash value to solve the conflict. dictionary: typedef struct dict {// type-specific function dictType * type; // Private Data void * privdata; // hash table dictht ht [2]; // rehash index in trehashidx ;} dict;

Integer Set

• An integer set is one of the underlying implementations of a set key. When a set contains only integer numeric elements and the number of elements in the set is small, Redis uses the integer set as the underlying implementation of the set key. For example, if we create a set key that contains only five elements and all elements in the set are integers, the underlying implementation of the set key will be an integer set.

Typedef struct intset {// encoding method uint32_t encoding; // number of elements contained in the Set uint32_t length; // The int8_t contents [];} intset; the contents array is the underlying implementation of the Integer Set: each element of the Integer Set is any item (element) in the contents array ), the length attribute of each item sorted in ascending order by value in the array records the number of elements contained in the Integer Set, which is also the length of the contents array. Although the intset structure declares the contents attribute as the int8_t type, the contents array does not actually store any int8_t type value. The actual type of the contents array depends on the value of the encoding attribute. This is also one of the characteristics of the Integer Set, dynamic change of the contents size

• Upgrade
  Every time we add a new element to the Integer Set, and the type of the new element is longer than the type of all existing elements in the Integer Set, the integer set must be upgraded first, then add the new element to the Integer Set.
  1. Expand the space of the underlying array of the Integer Set Based on the type of the new element and allocate space for the new element.
  2. convert all existing elements in the underlying array to the same type as the new element, and place the converted elements in the correct position, the ordered nature of the underlying array remains unchanged.
  3. Place the new elements in the underlying array.

The integer set does not support downgrading.

Compressed list

• Compressing a list is one of the underlying implementations of the list key and hash key. When a list key contains only a few list items, and each list item is either a small integer or a short string, redis uses the compressed list as the underlying implementation of the list key.

• In addition, when a hash key only contains a few key-value pairs, and each key-value pair is either a small integer or a short string, redis uses the compressed list as the underlying implementation of the hash key.

• The compressed list is developed by Redis to save memory. It is an ordered data structure consisting of a series of specially encoded continuous memory blocks. A compressed list can contain any number of nodes, each node can store an array of bytes or an integer.

• Each Compressed Column node can save an array of bytes or an integer. Each compressed list node consists of three parts: previous_entry_length, econding, and content.
  1. previus_entry_length records the length of the previous node in the compression list. Because the previus_entry_length attribute of the node records the length of the previous node, the program can calculate the start address of the previous Node Based on the start address of the current node through pointer operation.
  2. The encoding attribute records the type and length of the data saved by the content attribute of the node.
  3. The content attribute stores the node value. The Node value can be a byte array or integer. the type and length of the value are determined by the node's encoding attribute.

Attribute	Type	Byte Length	Purpose
Zlbytes	Uint32_t	4	Record the memory bytes occupied by the entire compression list: Used to re-allocate the compressed list or calculate the zlend location
Zltail	Uint32_t	4	Record the number of bytes from the start address of the compressed list at the end of the compressed list: With this offset, the program can determine the address of the End Node of the table without having to traverse the compression list.
Zllen	Uint16_t	2	The number of nodes contained in the compression list is recorded: when the value of this attribute is uint16_max, the value of this attribute is the number of nodes contained in the compression list; when this value is equal to uint16_max, the actual number of nodes needs to traverse the entire compression list for calculation.
EntryX	List nodes	Indefinite	Each node in the compressed list. The node length is determined by the content saved by the node.
Zlend	Uint8_t	1	Special Value 0xFF (decimal 255), used to mark the end of the compressed list

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.