Redis code structure Class 3 database-list

Redis code structure Class 3 database-list

1. redis_list (t_list.c)

Commands of this type include lpush, rpush, lpop, and rpop. Here we will only introduce the lpush command. The corresponding command callback function is void lpushcommand (redisclient * c) {pushgenericcommand (C, redis_head);} Let's look at pushgenericcommand directly.

Void pushgenericcommand (redisclient * C, int where) {Int J, addlen = 0, pushed = 0; robj * lobj = lookupkeywrite (c-> dB, c-> argv [1]); // check whether the key exists. If the key exists, its subobject int may_have_waiting_clients = (lobj = NULL) is returned ); // The key does not exist if (lobj & lobj-> type! = Redis_list) {// if the key exists, its type must be redis_list addreply (C, shared. wrongtypeerr); Return ;}for (j = 2; j <c-> argc; j ++) {c-> argv [J] = tryobjectencoding (c-> argv [J]); If (may_have_waiting_clients) {If (handleclientswaitinglistpush (C, C-> argv [1], c-> argv [J]) {// determine whether a block pop key exists. This is caused by blpop addlen ++; continue ;} else {may_have_waiting_clients = 0 ;}} if (! Lobj) {// if the key does not have a value, call lobj = createziplistobject () only once. // always use ziplist and assign values at the beginning, as you can see, ziplist is actually a string array dbadd (c-> dB, C-> argv [1], lobj); // set the key, robj value added to dict} listtypepush (lobj, C-> argv [J], where); // Add each value to the value list, and this function will determine whether to convert the storage type pushed ++;} addreplylonglong (C, addlen + (lobj? Listtypelength (lobj): 0); If (pushed) signalmodifiedkey (c-> dB, C-> argv [1]); server. Dirty ++ = pushed ;}

The values above are a bit awkward. In fact, all the values obtained through the key are a robj value, which contains a list or ziplist, which is composed of multiple robj value objects. Let's take a look at the listtypepush function:

Void listtypepush (robj * subject, robj * value, int where) {/* check if we need to convert the ziplist */listtypetryconversion (subject, value ); // check whether ziplist needs to be converted to a common list_max_ziplist_entries file. // If the ziplist file is of the ziplist type and has a length greater than the configured list_max_ziplist_entries file, then, the ziplist is converted to the publish list if (subject-> encoding = redis_encoding_ziplist & ziplistlen (subject-> PTR)> = server. list_max_ziplist_entries) listtypeconvert (subject, redis_encodi Ng_shortlist); If (subject-> encoding = redis_encoding_ziplist) {// you can still use ziplist int Pos = (where = redis_head )? Ziplist_head: ziplist_tail; value = getdecodedobject (value); // obtain the string-type subject-> PTR = ziplistpush (subject-> PTR, value-> PTR, sdslen (value-> PTR), POS); // insert decrrefcount (value) in ziplist;} else if (subject-> encoding = redis_encoding_shortlist) {// list type, this is our common bidirectional linked list insertion. Here we will not explain if (where = redis_head) {listaddnodehead (subject-> PTR, value);} else {listaddnodetail (subject-> PTR, value);} incrrefcount (Value) ;}else {redispanic ("unknown list encoding") ;}// conversion type, if the length of the value to be saved is greater than the configured list_max_ziplist_value, you must convert the original ziplist to listvoid listtypetryconversion (robj * subject, robj * value) {If (subject-> encoding! = Dependencies) return; If (value-> encoding = redis_encoding_raw & sdslen (value-> PTR)> server. list_max_ziplist_value) listtypeconvert (subject, redis_encoding_linkedlist );}

Through the above code, we can find that when the length of the value to be inserted (actual byte length) is greater than the configured list_max_ziplist_value, or the ziplist length (element size) is greater than the server. list_max_ziplist_entries converts ziplist to list type. Next we will mainly introduce ziplist operations.

1.1 ziplist

/* Create a New empty ziplist. */unsigned char * ziplistnew (void) {unsigned int bytes = ziplist_header_size + 1; unsigned char * ZL = zmalloc (bytes); ziplist_bytes (zl) = bytes; bytes (zl) = ziplist_header_size; ziplist_length (zl) = 0; ZL [bytes-1] = zip_end; return ZL;} // Add an entry to ziplist. Zl is the ziplist string array; S is the actual content of the value to be inserted; slen is the value length, where is the first or last unsigned char * ziplistpush (unsigned char * ZL, u Nsigned char * s, unsigned int slen, int where) {unsigned char * P; // If P is the actual position of the inserted head, it starts from the header, is tail starting from the last byte P = (where = ziplist_head )? Ziplist_entry_head (zl): ziplist_entry_end (zl); Return _ ziplistinsert (zl, P, S, slen );}

The above new shows the ziplist structure: ziplist is a string array in the format:
[Header: <zlbytes> <zltail> <zllen>] [Body: <entry>... <Entry>] [end: <zlend>]. The first three are called ziplist headers.
Zlbytes: uint32_t, which stores the total length of the string array, including Header, body, and end;
Zltail: uint32_t, used to save the offset of the last entry, that is, the position of the last node. The pop operation is allowed without traversing.
Zllen: uint16_t, used to save the number of ziplist entries, so a maximum of 2 ^ 16-1 entries can be created.
Zlend: 1byte end mark [255]
Next let's take a look at the body: the structure of the entry: (each body is also composed of three parts)
Prevrawlen: stores the length of the previous node (this Len is the total length of an entry), which occupies 1 or 5 bytes. When Len is less than 254, it occupies 1 byte, otherwise, the first byte is saved as 254, and the last four bytes are saved as the actual length.
Lensize [curencode | curlen]: stores the encode type used by the current node. If it is a string type, it must also include the length. If it is an integer type, it is not required because the length of the integer type is fixed. This field may occupy 1, 2, 5, 1, and 1 bytes. When the content cannot be converted to long, the string must be used for encode. If the length of the string is smaller than 63 (2 ^ 6-1), zip_str_06b can be used for encode, this field occupies 1 byte. The first two digits indicate zip_str_06b (00), and the last six digits indicate the actual length. If the length of the string is less than 16384 (2 ^ 14-1 ), zip_str_14b can be used for encode. This field occupies 2 bytes. The first two digits indicate zip_str_14b (01), and the last 14 digits indicate the actual length. If the length of the string is greater than or equal to 16384, zip_str_32b is used for encode. This field occupies five bytes. The first byte indicates zip_str_32b (,), and the last four bytes (32 bits) indicate the actual length;
If the content can be converted to long, it is converted to int16_t, int32_t, int64_t of the corresponding type. In this case, this field only occupies one byte, the four bits in height indicate the encode type, respectively (1110,). At this time, the length of the content is no longer required because the size of each int may be obtained by the type.
Value: stores the actual content, which is stored according to the preceding encode. The length used. If it is a string, it is determined by the length of the string. If it is an integer, it is determined by each type (2, 4, 8 ). For example:

Figure 1 ziplist Structure

Next let's take a look at how ziplist is inserted:

Static zlentry zipentry (unsigned char * P) {zlentry E; E. prevrawlen = zipprevdecodelength (p, & E. prevrawlensize); // returns the total length of the previous node, and saves the number of bytes occupied by the stored length value to E. prevrawlensize, that is, 1 or 5 (see the preceding description) E. len = zipdecodelength (p + E. prevrawlensize, & E. lensize); // return the length of the content of the current node, and save the size occupied by lensize to E. lensize, that is, E. headersize = E. prevrawlensize + E. lensize; // the length of the first two fields of each node. len is the actual length of the node. encoding = zipentryencod Ing (p + E. prevrawlensize); // return the encode e of the current node. P = P; Return e;} // ZL is the ziplist to be operated, and P is the position of the current operation. The value varies depending on head | tail, if it is head, it is the position after ziplist: Header; if it is tail, it is in ziplist: end: <zlend>, and s is the actual content, slen is the actual length of static unsigned char * _ ziplistinsert (unsigned char * ZL, unsigned char * P, unsigned char * s, unsigned int slen) {size_t curlen = ziplist_bytes (zl ), reqlen, prevlen = 0; size_t offset; int nextdiff = 0; unsigned char Encoding = 0; long value; zlentry entry, tail;/* Find Out prevlen for the entry that is inserted. */If (P [0]! = Zip_end) {// This indicates from the header insert entry = zipentry (p); // see the preceding comment prevlen = entry. prevrawlen; // return the length of the previous node. If you insert data from the head, it is obviously 0} else {// insert unsigned char * ptail = ziplist_entry_tail (zl) from tail ); // obtain the starting position of the last node if (ptail [0]! = Zip_end) {// if the current list is not empty prevlen = ziprawentrylength (ptail ); // obtain the total space of the last node}/* See if the entry can be encoded calculate the actual size of the reqlen bytes occupied by storing the content for the node. */If (ziptryencoding (S, slen, & Value, & encoding) {// integer 2, 4, 8/* 'encoding' is set to the appropriate integer encoding */reqlen = zipintsize (encoding );} else {// the actual length of the string/* 'encoding' is untouched, however zipencodelength will use the * string Length to figure out how to encode it. */reqlen = slen;}/* We need space for both the length of the previous entry and * the length of the payload. */reqlen + = zipprevencodelength (null, prevlen); // The amount of space required to save the length of the previous node, that is, the first field of the body occupies space 1 or 5 reqlen + = zipencodelength (null, encoding, slen); // the space occupied by the second field of the Body 1, 2, 5, 1, 1, 1/* When the insert position is not equal to the tail, we need to * Make sure that th E next entry can hold this entry's length in * its prevlen field. */nextdiff = (P [0]! = Zip_end )? Zipprevlenbytediff (p, reqlen): 0; // if it is head insert, it is clear that if the prevrawlen of the header node is changed at this time, and when the length of the node to be inserted is greater than 245, you need to add four bytes. The original header is empty, so only one byte is saved as 0, now, we need five bytes to save the length of the newly inserted node/* store offset because a realloc may change the address of ZL. */offset = p-ZL; ZL = ziplistresize (zl, curlen + reqlen + nextdiff); // The re-allocated memory reqlen is the length of the currently inserted node, nextdiff is the length to be added P = ZL + offset;/* apply memory move when necessary and update tail offset. */If (P [0]! = Zip_end) {// header insert. Obviously, the previous content is still in front, so you need to move them back./* subtract one because of the zip_end bytes */memmove (p + reqlen, p-nextdiff, curlen-offset-1 + nextdiff); // here nextdiff is used to indicate that this nextdiff is also the content of the previous node/* encode this entry's raw length in the next entry. */zipprevencodelength (p + reqlen, reqlen); // update prevlen/* update offset for tail */ziplist_tail_offset (zl) + = reqlen; /* When the tail contai NS more than one entry, we need to take * "nextdiff" in account as well. otherwise, a change in the * size of prevlen doesn't have an effect on the * tail * offset. */tail = zipentry (p + reqlen); If (P [reqlen + tail. headersize + tail. len]! = Zip_end) ziplist_tail_offset (zl) + = nextdiff;} else {// tail insert/* this element will be the new tail. */ziplist_tail_offset (zl) = p-ZL;}/* When nextdiff! = 0, the raw length of the next entry has changed, so * We Need to cascade the update throughout the ziplist */If (nextdiff! = 0) {// if the newly inserted node Len is greater than 254, that is, when the prevlen of the previous node needs to be extended to five nodes for storage, you need to traverse the entire list in sequence to modify the prevlen of each node. The current traversal may not be very long, because four bytes are added from the head, when a prevlen + 4 is smaller than 254, the traversal ends offset = p-ZL; ZL = _ ziplistcascadeupdate (zl, P + reqlen ); P = ZL + offset;}/* write the entry */P + = zipprevencodelength (p, prevlen ); // Save the prevlen P + = zipencodelength (p, encoding, slen) of the newly inserted node; // Save the lensize if (zip_is_str (encoding) of the newly inserted node )) {// Save the actual content memcpy (P, S, slen);} else {zipsaveinteger (p, value, encoding);} ziplist_incr_length (zl, 1); Return ZL ;}

All in all, ziplist is essentially a string array. It stores the size of the last node of each node, the encode of the current nodule, and the actual length to achieve traversal in two aspects. In addition, from the above operation, we can see that when inserting from the head, the entire insert requires more processes: memmove, and the extended value nextdiff is calculated, which leads to the change of prevlen of the entire list over time. However, both the head and tail require a realloc process, so although ziplist can save memory, it uses CPU for mem. If you want to reduce the usage, you can modify the two configurations mentioned above. And try to use rpush for better performance.