Ziplist looks similar to the adlist name I previously resolved, but it works completely differently. The previous adlist mainly targets common data linked list operations. Today's ziplist refers to a compressed linked list. Why is it called a compressed linked list? Because pre and next are commonly used in the linked list to indicate the previous pointer of the current node or
Ziplist looks similar to the adlist name I previously resolved, but it works completely differently. The previous adlist mainly targets common data linked list operations. Today's ziplist refers to a compressed linked list. Why is it called a compressed linked list? Because pre and next are commonly used in the linked list to indicate the previous pointer of the current node or
Ziplist looks similar to the adlist name I previously resolved, but it works completely differently. The previous adlist mainly targets common data linked list operations. Today's ziplist refers to a compressed linked list. Why is it called a compressed linked list? Because pre is usually used in linked lists, next indicates the previous pointer of the current node or the next pointer of the current node. This actually occupies a large amount of memory space to a certain extent, ziplist uses the length representation method. The whole ziplist is actually a super long string. Through the length of each node in it, the length of the previous node, and other information, you can quickly locate and implement related operations, in addition, the writer also dynamically allocates bytes in length to indicate the length, avoiding certain memory consumption. For example, each character string of a node is short, however, when you use several bytes to represent the length of a string, this obviously results in a lot of waste. Therefore, ziplist achieves compression in terms of length representation, which also reflects the compression performance. Where is ziplist used? ziplist is the method used to add data to the Linked List, such as rpush and lpush, which are commonly used. Then we will see the corresponding implementation method.
At the beginning of learning ziplist, we must understand its structure. It takes some time to think about it. Otherwise, it is not easy to understand the design of the White House. Below is my understanding to help you understand:
/* The ziplist is a specially encoded dually linked list that is designed * to be very memory efficient. it stores both strings and integer values, * where integers are encoded as actual integers instead of a series of * characters. it allows push and pop operations on either side of the list * in O (1) time. however, because every operation requires a reallocation of * the memory used by the ziplist, the actual complexity is related to the * amount of memory used by the ziplist. ** ziplist is an encoded list. It is specially designed to make memory operations very efficient. This list can store * string and integer types at the same time, the list can support the push and pop-up operations at the beginning and end of the O (1) constant time, but because each * Operation is designed to re-allocate and release the memory, therefore, The operation complexity is increased. * the ziplist structure consists of * ziplist overall layout: * The general layout of the ziplist is as follows :*
**
Is an unsigned integer to hold the number of bytes that the * ziplist occupies. This value needs to be stored to be able to resize the * entire structure without the need to traverse it first .*
Indicates the number of bytes occupied by the ziplist. Therefore, you do not need to traverse the ziplist from the beginning when you resize it **
Is the offset to the last entry in the list. This allows a pop * operation on the far side of the list without the need for full traversal .*
The position of the last entry is recorded in the list, so that operations can be conveniently popped up at the end of the list **
Is the number of entries. When this value is larger than 2 ** 16-2, * we need to traverse the entire list to know how many items it holds .*
Records the total number of entry data nodes in ziplist **
Is a single byte special value, equal to 255, which indicates the * end of the list .*
Indicates the end identifier, expressed in a single byte. The value is 255, Which is 11111111 ** ziplist entries: * Every entry in the ziplist is prefixed by a header that contains two pieces * of information. first, the length of the previous entry is stored to be * able to traverse the list from back to front. second, the encoding with an * optional string length of the entry itself is stored. * Each entry data node mainly contains two parts of information. The first one is the length of the previous node, and the whole list can be traversed from any node from the back. * The second one, encoding string type saving ** The length of the previous entry is encoded in the following way: * If this length is smaller than 254 bytes, it will only consume a single * byte that takes the length as value. when the length is greater than or * equal to 254, it will consume 5 bytes. the first byte is set to 254 to * indicate a larger value is following. the remaining 4 bytes take the * length of the previous entry as value. * If the length of a string at a previous data node is less than 254 bytes, it will consume 8 bytes in a single byte, the maximum value can be 2 to the power of 8 * When the length of a string is greater than 254 bytes, it is expressed in 5 bytes. The first byte is set to 254, the remaining four bytes occupy the length of the previous data node. ** the other header field of The entry itself depends on the contents of the * entry. when the entry is a string, the first 2 bits of this header will hold * the type of encoding used to store the length of the string, followed by the * actual length of the string. when the entry is an integer the first 2 bits * are both set to 1. the following 2 bits are used to specify what kind of * integer will be stored after this header. an overview of the different * types and encodings is as follows: * another value in the header information records the encoding method. When the encoding is a string, the first two digits of the header are, 10 three types in total * If the encoding is an integer number, the first two digits of the header are 11, representing the integer encoding, the next two digits indicate what type of integer value will be encoded * 00-int16_t, 01-int32_t, 10-int64_t, 11-24 bit signed behind the header. There are two special values, 11111110-8 bit signed, * 1111 0000-1111 1101, which indicates that the integer value 0-12 already exists and cannot be used, with the traditional representation of the length through a fixed pointer, benefits of doing so * more reasonable memory allocation ** three String encoding formats * | 00pppppp |-1 byte * String value with length less than or equal to 63 bytes (6 bits ). * | 01pppp | qqqqqqqq |-2 bytes * String value with length less than or equal to 16383 bytes (14 bits ). * | 10 ______ | qqqqqqqq | rrrrrrrr | ssssssss | tttttttt |-5 bytes * String value with length greater than or equal to 16384 bytes. * | 11000000 |-1 byte * Integer encoded as int16_t (2 bytes ). * | 11010000 |-1 byte * Integer encoded as int32_t (4 bytes ). * | 11100000 |-1 byte * Integer encoded as int64_t (8 bytes ). * | 11110000 |-1 byte * Integer encoded as 24 bit signed (3 bytes ). * | 11111110 |-1 byte * Integer encoded as 8 bit signed (1 byte ). * | 1111xxxx |-(with xxxx between 0000 and 1101) immediate 4 bit integer. * Unsigned integer from 0 to 12. the encoded value is actually from * 1 to 13 because 0000 and 1111 can not be used, so 1 shocould be * subtracted from the encoded 4 bit value to obtain the right value. * | 11111111 |-End of ziplist. ** All the integers are represented in little endian byte order. **----------------------------------------------------------------------------
I hope you can read it carefully and understand the author's design ideas. The following describes the definition of his actual struct:
/* Actual data storage node */typedef struct zlentry {// prevrawlen is the length of the previous data node, and prevrawlensize is the number of bytes required to record this length value. unsigned int prevrawlensize, prevrawlen; // len indicates the length of the current data node. lensize indicates the current length and the number of bytes. unsigned int lensize, len; // number of bytes of the header information length of the data node unsigned int headersize; // The encoding method unsigned char encoding; // the data of the Data Node (including the header information ), save unsigned char * p;} zlentry as a string ;/*
Structure line representation
(Length of the previous node)
(Encoding method of the current node and length of the encoded data)
(Encoding data of the current node )*/
Let's take a look at the core operations and insert operations, which involve various back-and-forth movements of pointers. These are adjustments to the memory address:
/* Insert item at "p ". * // * insert operation implementation */static unsigned char * _ ziplistInsert (unsigned char * zl, unsigned char * p, unsigned char * s, unsigned int slen) {size_t curlen = intrev32ifbe (ZIPLIST_BYTES (zl), reqlen; unsigned int prevlensize, prevlen = 0; size_t offset; int nextdiff = 0; unsigned char encoding = 0; long value = 123456789;/* initialized to avoid warning. using a value that is easy See if for some reason we use it uninitialized. */zlentry tail;/* Find out prevlen for the entry that is inserted. * /// find the inserted position if (p [0]! = ZIP_END) {// locate the specified position ZIP_DECODE_PREVLEN (p, prevlensize, prevlen);} else {// If the inserted position is the end node, directly locate the end node, looking at the first byte, you can determine unsigned char * ptail = ZIPLIST_ENTRY_TAIL (zl); if (ptail [0]! = ZIP_END) {prevlen = zipRawEntryLength (ptail) ;}}/* See if the entry can be encoded */if (zipTryEncoding (s, slen, & value, & encoding )) {/* 'encoding' is set to the appropriate integer encoding */reqlen = zipIntSize (encoding);} else {/* 'encoding' is untouched, however zipEncodeLength will use the * string length to figure out how to encode it. */reqlen = slen;}/* We need space for both Length of the previous entry and * the length of the payload. */reqlen + = zipPrevEncodeLength (NULL, prevlen); reqlen + = zipEncodeLength (NULL, encoding, slen);/* When the insert position is not equal to the tail, we need to * make sure that the next entry can hold this entry's length in * its prevlen field. */nextdiff = (p [0]! = ZIP_END )? ZipPrevLenByteDiff (p, reqlen): 0;/* Store offset because a realloc may change the address of zl. * /// adjust the size of the reserved space for the insertion of the new node offset = p-zl; zl = ziplistResize (zl, curlen + reqlen + nextdiff); p = zl + offset; /* Apply memory move when necessary and update tail offset. */if (p [0]! = ZIP_END) {/* Subtract one because of the ZIP_END bytes * // If the inserted position is not the end node, move memmove (p + reqlen, p-nextdiff, curlen-offset-1 + nextdiff);/* Encode this entry's raw length in the next entry. */trim (p + reqlen, reqlen);/* Update offset for tail */ZIPLIST_TAIL_OFFSET (zl) = intrev32ifbe (intrev32ifbe (ZIPLIST_TAIL_OFFSET (zl) + reqlen ); /* When the tail contains more than one entry, we need To take * "nextdiff" in account as well. otherwise, a change in the * size of prevlen doesn't have an effect on the * tail * offset. */tail = zipEntry (p + reqlen); if (p [reqlen + tail. headersize + tail. len]! = ZIP_END) {ZIPLIST_TAIL_OFFSET (zl) = intrev32ifbe (intrev32ifbe (ZIPLIST_TAIL_OFFSET (zl) + nextdiff) ;}} else {// if it is the end node, directly set the new tail node/* This element will be the new tail. */ZIPLIST_TAIL_OFFSET (zl) = intrev32ifbe (p-zl);}/* When nextdiff! = 0, the raw length of the next entry has changed, so * we need to cascade the update throughout the ziplist */if (nextdiff! = 0) {offset = p-zl; zl = _ ziplistCascadeUpdate (zl, p + reqlen); p = zl + offset ;} /* Write the entry * // Write new data node information p + = zipPrevEncodeLength (p, prevlen); p + = zipEncodeLength (p, encoding, slen ); if (ZIP_IS_STR (encoding) {memcpy (p, s, slen);} else {zipSaveInteger (p, value, encoding );} // Add 1 ZIPLIST_INCR_LENGTH (zl, 1); return zl;} to the length of the update list ;}The delete operation is as follows:
/* Delete "num" entries, starting at "p ". returns pointer to the ziplist. * // * The deletion method involves the sliding of the p pointer. The following addresses must be moved */static unsigned char * _ ziplistDelete (unsigned char * zl, unsigned char * p, unsigned int num) {unsigned int I, totlen, deleted = 0; size_t offset; int nextdiff = 0; zlentry first, tail; first = zipEntry (p ); for (I = 0; p [0]! = ZIP_END & I <num; I ++) {p + = zipRawEntryLength (p); deleted ++ ;}totlen = p-first.p; if (totlen> 0) {if (p [0]! = ZIP_END) {/* Storing 'prevrawlen' in this entry may increase or decrease the * number of bytes required compare to the current 'prevrawlen '. * There always is room to store this, because it was previusly * stored by an entry that is now being deleted. */nextdiff = zipPrevLenByteDiff (p, first. prevrawlen); p-= nextdiff; zipPrevEncodeLength (p, first. prevrawlen);/* Update offset for tail */ZIPLIS T_TAIL_OFFSET (zl) = intrev32ifbe (intrev32ifbe (ZIPLIST_TAIL_OFFSET (zl)-totlen);/* When the tail contains more than one entry, we need to take * "nextdiff" in account as well. otherwise, a change in the * size of prevlen doesn't have an effect on the * tail * offset. */tail = zipEntry (p); if (p [tail. headersize + tail. len]! = ZIP_END) {ZIPLIST_TAIL_OFFSET (zl) = intrev32ifbe (intrev32ifbe (ZIPLIST_TAIL_OFFSET (zl) + nextdiff);}/* Move tail to the front of the ziplist */memmove (first. p, p, intrev32ifbe (ZIPLIST_BYTES (zl)-(p-zl)-1);} else {/* The entire tail was deleted. no need to move memory. */ZIPLIST_TAIL_OFFSET (zl) = intrev32ifbe (first. p-zl)-first. prevrawlen);}/* Resize and update length * // adjust the list size offset = fi Rst. p-zl; zl = ziplistResize (zl, intrev32ifbe (ZIPLIST_BYTES (zl)-totlen + nextdiff); ZIPLIST_INCR_LENGTH (zl,-deleted); p = zl + offset; /* When nextdiff! = 0, the raw length of the next entry has changed, so * we need to cascade the update throughout the ziplist */if (nextdiff! = 0) zl = _ ziplistCascadeUpdate (zl, p);} return zl ;}This method is used to delete num nodes starting from the node corresponding to the index. This is the most primitive method to delete, and other methods are packaged for this method.
Next, let's take a look at the methods we entered in the redis command line to call lpush or rpush? Call method:
zl = ziplistPush(zl, (unsigned char*)"foo", 3, ZIPLIST_TAIL); zl = ziplistPush(zl, (unsigned char*)"quux", 4, ZIPLIST_TAIL); zl = ziplistPush(zl, (unsigned char*)"hello", 5, ZIPLIST_HEAD);
/* Method for inserting data at the two sides of the list */unsigned char * ziplistPush (unsigned char * zl, unsigned char * s, unsigned int slen, int where) {unsigned char * p; // start to directly locate p = (where = ZIPLIST_HEAD) here )? ZIPLIST_ENTRY_HEAD (zl): ZIPLIST_ENTRY_END (zl); // call the insert method return _ ziplistInsert (zl, p, s, slen) after the data is inserted );}Finally, the insert method is called. I have read some ziplist analyses analyzed by others before writing them. I feel that some of them are very rough. I think it is clear that I have read the source code carefully. Each person has a different focus. Finally, the header file and key macro definitions are given:
/* The end value of the zip list */# define ZIP_END 255/* Maximum length of the zip list */# define ZIP_BIGLEN 254/* Different encoding/length possibilities * // * Different codes */# define ZIP_STR_MASK 0xc0 # define ZIP_INT_MASK 0x30 # define ZIP_STR_06B (0 <6) # define ZIP_STR_14B (1 <6) # define ZIP_STR_32B (2 <6) # define ZIP_INT_16B (0xc0 | 0 <4) # define ZIP_INT_32B (0xc0 | 1 <4) # define ZIP_INT_64B (0xc0 | 2 <4) # define ZIP_INT_24B (0xc0 | 3 <4) # define ZIP_INT_8B 0xfe/* 4 bit integer immediate encoding */# define ZIP_INT_IMM_MASK 0x0f // subsequent operations require bitwise operations with the mask # define zip_int_imm_0xmin 0xf1/* 11110001 */# define ZIP_INT_IMM_MAX 0xfd/* 11111101 * // The maximum value cannot be 11111111, this is the same as the last node # define ZIP_INT_IMM_VAL (v) (v & ZIP_INT_IMM_MASK) # define INT24_MAX 0x7fffff # define INT24_MIN (-INT24_MAX-1) /* Macro to determine type */# define ZIP_IS_STR (enc) & ZIP_STR_MASK) <ZIP_STR_MASK) /* Utility macros * // * below are some numerical offsets that can be directly located at that time */# define ZIPLIST_BYTES (zl) (* (uint32_t *) (zl ))) # define ZIPLIST_TAIL_OFFSET (zl) (* (uint32_t *) (zl) + sizeof (uint32_t) # define ZIPLIST_LENGTH (zl) (* (uint16_t *) (zl) + sizeof (uint32_t) * 2) # define ZIPLIST_HEADER_SIZE (sizeof (uint32_t) * 2 + sizeof (uint16_t) # define limit (zl) + attributes) # define attributes (zl) + intrev32ifbe (ZIPLIST_TAIL_OFFSET (zl) # define attributes (zl) + intrev32ifbe (ZIPLIST_BYTES (zl )) -1)
. H file:
/** Copyright (c) 2009-2012, Pieter Noordhuis
* Copyright (c) 2009-2012, Salvatore Sanfilippo * All rights reserved. ** Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * ** Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclawing. ** Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclawing in the * documentation and/or other materials provided with the distribution. ** Neither the name of Redis nor the names of its contributors may be used * to endorse or promote products derived from this software without * specific prior written permission. ** this software is provided by the copyright holders and contributors "as is" * and any express or implied warranties, INCLUDING, but not limited, THE * implied warranties of merchantability and fitness for a special PURPOSE * are disclaimed. in no event shall the copyright owner or contributors be * liable for any direct, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * consequential damages (INCLUDING, but not limited, procurement of * substitute goods or services; loss of use, DATA, or profits; or business * INTERRUPTION) however caused and on any theory of liability, whether in * CONTRACT, strict liability, or tort (including negligence or otherwise) * arising in any way out of the use of this software, even if advised of the * possibility of such damage. * // * mark the ID of the first and last nodes in the list */# define ZIPLIST_HEAD 0 # define ZIPLIST_TAIL 1 unsigned char * ziplistNew (void ); // create a new list of unsigned char * ziplistPush (unsigned char * zl, unsigned char * s, unsigned int slen, int where ); // push the data unsigned char * ziplistIndex (unsigned char * zl, int index) into the image list; // The index locates at a location in the list unsigned char * ziplistNext (unsigned char * zl, unsigned char * p); // obtain the next value of the position in the current list. unsigned char * ziplistPrev (unsigned char * zl, unsigned char * p ); // obtain the previous value of the current list position unsigned int ziplistGet (unsigned char * p, unsigned char ** sval, unsigned int * slen, long * lval ); // obtain the list information unsigned char * ziplistInsert (unsigned char * zl, unsigned char * p, unsigned char * s, unsigned int slen ); // Insert the unsigned char * ziplistDelete (unsigned char * zl, unsigned char ** p) into the List ); // delete a node unsigned char * ziplistDeleteRange (unsigned char * zl, unsigned int index, unsigned int num) in the list; // start from the node corresponding to the index, delete num nodes unsigned int ziplistCompare (unsigned char * p, unsigned char * s, unsigned int slen); // comparison method between lists unsigned char * ziplistFind (unsigned char * p, unsigned char * vstr, unsigned int vlen, unsigned int skip); // search for a node unsigned int ziplistLen (unsigned char * zl) in the list ); // return the length of the list size_t ziplistBlobLen (unsigned char * zl); // return the binary length of the list, which returns the number of bytes