Redis design and implementation-data structure and object, redis Data Structure

Redis design and implementation-data structure and object, redis Data Structure
1. simple dynamic string -- simple dynamic string implementation

Relative to the C string

1. Get the string length with constant complexity

2. Prevent Buffer Overflow

3. Reduce the number of times the memory is re-allocated when the string is modified (Space pre-allocation and inert space allocation)

4. Binary Security (not only can save text data, but also can save binary data in any format)

5. compatible with some C string functions

2 Linked List Implementation

List listNode

Features

Double-ended, no-ring, table head pointer and table tail pointer, with chain table length counter, Polymorphism

3 dictionary usage

Underlying Implementation of the hash key in the Redis Database

Data Structure

Hash Table dictht hash table node dictEntry dictionary dict

Hash Algorithm

MurmurHash2 Algorithm

Key conflict

Add a new node to the table header of the linked list

Rehash Process

1) Allocate space for ht [1]

2) rehashindex = 0;

3) Each CURD, rehash all key-value pairs of ht [0] On the reindexhash index to ht [1], rehashindex ++;

4) When all key-value pairs are rehash to ht [1], rehashindex =-1.

4. Skip table -- skiplist time complexity

Best O (logN), worst O (N)

Usage

One of the underlying implementations of the sorted set key (when there are many elements in the ordered combination, or when the Members in the sorted set are long strings), the cluster node is used as the internal data structure.

Data Structure

Structure of zskiplistNode:

Typedef struct zskiplistNode {// layer struct zskiplistLevel {// forward pointer struct zskiplistNode * forward; // span unsigned int span;} level []; // return pointer struct zskiplistNode * backward; // double score; // member object robj * obj;} zskiplistNode;

Typedef struct zskiplist {// header node and table tail node struct zskiplistNode * header, * tail; // Number of table nodes unsigned long length; // int level for the node with the maximum number of layers in the table;} zskiplist;

5 Integer Set -- intset usage

A set contains only integer elements, and the number of elements in the set is small.

Complexity

The time complexity of adding new elements to an integer set is O (N)

Benefits

Improves flexibility and saves memory

Note:

Intset does not support downgrading

6. Compressed list -- ziplist usage

List key (only contains a few list items, and the list items are either small integer or short strings) and hash key (only contains a few key-value pairs, and the keys and values of each key-value pair are either small integers or short strings .)

Implementation

The length of the previus_entry_length attribute is one byte and five bytes, which records the length of the previous node. All programs can perform pointer operations, calculate the start address of the previous Node Based on the start address of the current node.

The encoding attribute records the type and length of the data stored in the content attribute, one byte, two bytes, five bytes. The maximum value is, 01, 10, which indicates that the byte array is saved, the other bits indicate the length of the byte array. The maximum bits of one byte is 11, indicating an integer. the type and length are determined by other bits.

Chain update

When a node is added and deleted, chained updates may be triggered.

The worst time complexity of chained update is O (n²), but the average time complexity of commands such as ziplistPush is O (N ).

7 Objects

Redis creates an object system based on the above data structure, including string objects, List objects, hash objects, set objects, and ordered set objects.

Redis's object system also realizes the memory recovery mechanism based on the reference counting technology, and implements the object sharing mechanism based on the reference counting technology, multiple database keys share the same object to save memory.

Every object in Redis is represented by a redisObject structure:

1 typedef struct redisObject {2 // type 3 unsigned type: 4; 4 5 // Code 6 unsigned encoding: 4; 7 8 // pointer to the underlying data structure 9 void * ptr; 10 11 //... 12} robj;

Refcount indicates reference count;

Lru records the last time the object was used by the application. The idling time is the value of the current time minus the lru value. If the maxmemory option is enabled on the server and the memory recovery algorithm is volatile-lru, allkeys-lru, when the memory usage exceeds the upper limit of maxmemory, the keys with high idling time will be preferentially released;

Type indicates the object type. The key is always a String object, and the value can be one of the five objects;

Encoding records the encoding used by the object,

7.1 String object 7.1.1 type and encoding

If the value is an integer and can be expressed as long, the encoding is set to int;

If a string is saved and the length is greater than 32 bytes, use SDS for saving and set the encoding to raw;

If a string is saved and the length is smaller than or equal to 32 bytes, the encoding is set to embstr;

7.1.2 Save the specific value and encoding

An integer that can be saved as long and encoded as int;

The floating point number that can be saved as long double type, encoded as embstr or raw;

The string value, or an integer that is too long to be saved by the long type, or a floating point that cannot be saved by the long double type, encoded as embstr or raw.

7.1.3 Conversion

In Redis, The embstr-encoded String object is read-only. Any modification to this object will convert it to a raw-encoded String object.

7.2 list object

The ziplist encoded list object uses the compressed list as the underlying implementation,

The list object encoded in the shortlist uses the double-ended list as the underlying implementation.

7.2.1 encoding conversion

Ziplist must meet the following requirements:

• All strings are less than 64 bytes in length. This value can be changed to list-max-ziplist-value.
• The number of elements is less than 512. This value can be changed to list-max-ziplist-entries.

7.3 hash object

The ziplist-encoded hash object uses the compressed list as the underlying implementation,

The hash object of hashtable encoding is implemented using the dictionary as the underlying layer. Each key value of the dictionary is a string object.

7.3.1 encoding conversion

Ziplist must meet the following requirements:

• The string length of all key-value pairs is smaller than 64 bytes. This value can be changed to hash-max-ziplist-value.
• The number of key-value pairs saved is less than 512. This value can be changed to hash-max-ziplist-entries.

7.4 collection object

Intset uses an integer set as the underlying implementation;

Hashtable uses dictionaries as the underlying implementation;

7.4.1 encoding and conversion

Intset must meet the following requirements:

• All elements contained in a collection object are integers;
• The number of elements contained in the set object cannot exceed 512. This value can be changed to set-max-ziplist-entries.

7.5 ordered collection object 7.5.1 ordered combination object of ziplist Encoding

The object uses the compressed list as the underlying implementation. Each set element is saved by two compressed list nodes, the first node stores the members of the elements, and the second node stores the score of the elements.

7.5.2 ordered combination objects of skiplist Encoding

The skiplist object uses the zset structure as the underlying implementation and contains both a dictionary and a skip table;

1 typedef struct zset {2     zskiplist *zsl;3     dict *dict;   4 } zset;

The zsl hop table stores all set elements from small to large by value. Through the Skip table, you can perform range operations on the ordered combination, such as zrank and zrange;

The dict dictionary stores the ing from members to scores. You can use O (1) Complexity to query the scores of a given Member, such as zscore.

Note: zsl and dict share the members and scores of the same element through pointers. No duplicate members or scores are generated and no additional memory is generated.

7.5.3 encoding conversion

Ziplist must meet the following requirements:

• The number of elements saved in an ordered set is less than 128. This value can be changed to zset-max-ziplist-entries.
• The length of all members stored in an ordered set is smaller than 64 bits. This value can be changed to zset-max-ziplist-value.