Redis source code analysis (7) --- zipmap compression Graph

If you have read the ziplist compressed list analysis I have analyzed, I think this is not a special problem. Ziplist compression lists and zipmap both use the dynamic byte allocation method to indicate the length. For example, the fixed byte representation saves a lot of space. The same problem is the complex pointer movement and character position movement. But overall

If you have read the ziplist compressed list analysis I have analyzed, I think this is not a special problem. Ziplist compression lists and zipmap both use the dynamic byte allocation method to indicate the length. For example, the fixed byte representation saves a lot of space. The same problem is the complex pointer movement and character position movement. But overall

If you have read the ziplist compressed list analysis I have analyzed, I think this is not a special problem. Ziplist compression lists and zipmap both use the dynamic byte allocation method to indicate the length. For example, the fixed byte representation saves a lot of space. The same problem is the complex pointer movement and character position movement. But in general, the advantage must be greater than the disadvantage, or the designer will not do the same. Ziplist stores a list. zipmap stores key-value pairs in the form of key: value. Below I will show the structure of zipmap, which is actually a super long string.

 
  
   "foo"
   
    
     "bar"
     
      "hello"
      
       
        "world" 
       
      
     
    
   
  
 

It involves several variables zmlen, len, and free. The complete explanation is given below:

/* String-> String Map data structure optimized for size. * This file implements a data structure mapping strings to other strings * implementing an O (n) lookup data structure designed to be very memory * efficient. ** The Redis Hash type uses this data structure for hashes composed of a small * number of elements, to switch to a hash table once a given number of * elements is reached. ** Given that frequently times Redis Hashes are used to represent objects composed * of few fields, this is a very big win in terms of used memory. ** zipmap compression tables are very similar to ziplist tables, both of which achieve high Memory Operation Efficiency ** Copyright (c) 2009-2010, Salvatore Sanfilippo * All rights reserved. ** Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * ** Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclawing. ** Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclawing in the * documentation and/or other materials provided with the distribution. ** Neither the name of Redis nor the names of its contributors may be used * to endorse or promote products derived from this software without * specific prior written permission. ** this software is provided by the copyright holders and contributors "as is" * and any express or implied warranties, INCLUDING, but not limited, THE * implied warranties of merchantability and fitness for a special PURPOSE * are disclaimed. in no event shall the copyright owner or contributors be * liable for any direct, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * consequential damages (INCLUDING, but not limited, procurement of * substitute goods or services; loss of use, DATA, or profits; or business * INTERRUPTION) however caused and on any theory of liability, whether in * CONTRACT, strict liability, or tort (including negligence or otherwise) * arising in any way out of the use of this software, even if advised of the * possibility of such damage. * // * Memory layout of a zipmap, for the map "foo" => "bar", "hello" => "world ":**
 
  
   
"Foo"
   
    
     
"Bar"
     
      
"Hello"
      
       
         "World "**
        
          Is 1 byte length that holds the current size of the zipmap. * When the zipmap length is greater than or equal to 254, this value * is not used and the zipmap needs to be traversed to find out the length. *
         
           It occupies 1 byte, so the maximum number of records that can be represented is 254. When the number of element records in zipmap exceeds this value, * the size can only be counted from the past to the next time, it is different from ziplist. **
          
            Is the length of the following string (key or value ).*
           
             Lengths are encoded in a single value or in a 5 bytes value. * If the first byte value (as an unsigned 8 bit value) is between 0 and * 252, it's a single-byte length. if it is 253 then a four bytes unsigned * integer follows (in the host byte ordering ). A value of 255 is used to * signal the end of the hash. the special value 254 is used to mark * empty space that can be used to add new key/value pairs. *
            
              Represents the length of the value of the string key or value. The length is generally encoded by 1 byte or 5 byte. This is similar to ziplist * if the length of the subsequent string is less than or equal to 252, it can be expressed in a single byte. Other lengths such as 253,254 are used to indicate other functions. When this number is exceeded *, the storage length is directly 5 bytes. **
             
               Is the number of free unused bytes after the string, resulting * from modification of values associated to a key. for instance if "foo" * is set to "bar", and later "foo" will be set to "hi ", it will have a * free byte to use if the value will enlarge again later, or even in * order to add a key/value pair if it fits. *
              
                Generally, it indicates the idle value of the value length. When key: value = "foo": "bar", it is changed to "foo": "hi ", the idle length is 1 **
               
                 Is always an unsigned 8 bit number, because if after an * update operation there are more than a few free bytes, the zipmap will be * reallocated to make sure it is as small as possible. *
                
                  Zipmap will adjust The size of zipmap to make the map as small as possible. ** The most compact representation of the above two elements hash is actually: * This is an example :*
                 
                  
                    "Foo"
                   
                    
                      "Bar"
                     
                       "Hello"
                      
                       
                         "World "*
                        <总键值对数>
                         <第一个key的长度>
                           Key character
                          <第一个value的长度>
                           <空闲长度开始都为0>
                             * "\ X02 \ x03foo \ x03 \ x00bar \ x05hello \ x05 \ x00world \ xff" ** Note that because keys and values are prefixed length "objects ", * the lookup will take O (N) where N is the number of elements * in the zipmap and * not * the number of bytes needed to represent the zipmap. * This lowers the constant times considerably. */
                           
                          
                         
                        
                       
                      
                     
                    
                   
                  
                 
                
               
              
             
            
           
          
         
        
       
      
     
    
   
  
 

When it comes to key-value pairs, the most important method is, of course, the key-value method. The method is as follows:

/* Set key to value, creating the key if it does not already exist. * If 'update' is not NULL, * update is set to 1 if the key was * already preset, otherwise to 0. */unsigned char * zipmapSet (unsigned char * zm, unsigned char * key, unsigned int klen, unsigned char * val, unsigned int vlen, int * update) {unsigned int zmlen, offset; unsigned int freelen, reqlen = zipmapRequiredLength (klen, vlen); unsign Ed int empty, vempty; unsigned char * p; freelen = reqlen; if (update) * update = 0; // find the key position p = zipmapLookupRaw (zm, key, klen, & zmlen); if (p = NULL) {/* Key not found: enlarge * // The Position of the key is not found. Adjust the zipmap size, prepare to add operation zm = zipmapResize (zm, zmlen + reqlen); p = zm + zmlen-1; zmlen = zmlen + reqlen;/* Increase zipmap length (this is an insert) * /// if the header byte has not reached the maximum value, it will increase progressively if (zm [0] <ZIPMAP_BIGLEN) zm [0] ++;} else {/* Key found. Is there enough space for the new value? * // * Compute the total length: */if (update) * update = 1; // locate and locate the key, determine whether there is space to insert a new value freelen = zipmapRawEntryLength (p); if (freelen <reqlen) {/* Store the offset of this key within the current zipmap, so * it can be resized. then, move the tail backwards so this * pair fits at the current position. * /// if there is no space to insert a new value, adjust the size offset = p-zm; zm = zipmapResize (zm, zmlen-freelen + reqlen); p = zm + offset; /* The + 1 in the number of bytes to be moved is caused by the * end-of-zipmap byte. note: the * original * zmlen is used. * // move the space to add a new value for memmove (p + reqlen, p + freelen, zmlen-(offset + freelen + 1); zmlen = zmlen-freelen + reqlen; freelen = reqlen;}/* We now have a suitable block where the key/value entry can * be written. if there is too much free space, move the tail * of the zipmap a few bytes to the front and shrink the zipmap, * as we want zipmaps to be very space efficient. */empty = freelen-reqlen; if (empty> = ZIPMAP_VALUE_MAX_FREE) {/* First, move the tail
 
  
Bytes to the front, then resize * the zipmap to be
  
   
Bytes smaller. */offset = p-zm; memmove (p + reqlen, p + freelen, zmlen-(offset + freelen + 1); zmlen-= empty; zm = zipmapResize (zm, zmlen); p = zm + offset; vempty = 0;} else {vempty = empty;}/* Just write the key + value and we are done. * // * Key: * // locate the inserted position. First, write the key value p + = zipmapEncodeLength (p, klen); memcpy (p, key, klen ); p + = klen;/* Value: * // The value behind the key Value is the value, and p + = zipmapEncodeLength (p, vlen) is written again; * p ++ = vempty; memcpy (p, val, vlen); return zm ;}
  
 

The method for returning the length in map is a bit special. If you locate it directly, you can use a byte to store the length:

/* Return the number of entries inside a zipmap * // * returns the map length */unsigned int zipmapLen (unsigned char * zm) {unsigned int len = 0; // if the first length is smaller than the maximum value, if (zm [0] <ZIPMAP_BIGLEN) {len = zm [0];} is returned directly. else {// otherwise the variable calculation length is unsigned char * p = zipmapRewind (zm); while (p = zipmapNext (p, NULL ))! = NULL) len ++;/* Re-store length if small enough */if (len <ZIPMAP_BIGLEN) zm [0] = len;} return len ;}

When we execute the set key "value" command on the redis client, we call the set method as follows:

    zm = zipmapSet(zm,(unsigned char*) "name",4, (unsigned char*) "foo",3,NULL);    zm = zipmapSet(zm,(unsigned char*) "surname",7, (unsigned char*) "foo",3,NULL);    zm = zipmapSet(zm,(unsigned char*) "age",3, (unsigned char*) "foo",3,NULL);

This method is much simpler than ziplist, And the header file is given at last.

/* String-> String Map data structure optimized for size. ** See zipmap. c for more info. ** ---------------------------------------------------------------------------- ** Copyright (c) 2009-2010, Salvatore Sanfilippo * All rights reserved. ** Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * ** Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclawing. ** Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclawing in the * documentation and/or other materials provided with the distribution. ** Neither the name of Redis nor the names of its contributors may be used * to endorse or promote products derived from this software without * specific prior written permission. ** this software is provided by the copyright holders and contributors "as is" * and any express or implied warranties, INCLUDING, but not limited, THE * implied warranties of merchantability and fitness for a special PURPOSE * are disclaimed. in no event shall the copyright owner or contributors be * liable for any direct, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * consequential damages (INCLUDING, but not limited, procurement of * substitute goods or services; loss of use, DATA, or profits; or business * INTERRUPTION) however caused and on any theory of liability, whether in * CONTRACT, strict liability, or tort (including negligence or otherwise) * arising in any way out of the use of this software, even if advised of the * possibility of such damage. */# ifndef _ ZIPMAP_H # define _ ZIPMAP_Hunsigned char * zipmapNew (void); // create a new compression chart unsigned char * zipmapSet (unsigned char * zm, unsigned char * key, unsigned int klen, unsigned char * val, unsigned int vlen, int * update); // sets the unsigned char * zipmapDel (unsigned char * zm, unsigned char * key, unsigned int klen, int * deleted); // delete a key-Value Pair unsigned char * zipmapRewind (unsigned char * zm) in the compression graph ); // It will be called in zipmapNext to unsigned char * zipmapNext (unsigned char * zm, unsigned char ** key, unsigned int * klen, unsigned char ** value, unsigned int * vlen); // gets the next key-value pair of this key-Value Pair int zipmapGet (unsigned char * zm, unsigned char * key, unsigned int klen, unsigned char ** value, unsigned int * vlen); // gets a key value pair int zipmapExists (unsigned char * zm, unsigned char * key, unsigned int klen ); // whether a key value in zipmap contains unsigned int zipmapLen (unsigned char * zm); // The total key value in the zipmap is size_t zipmapBlobLen (unsigned char * zm ); // The size void zipmapRepr (unsigned char * p) required for serialization of the compression graph into the file; // The output details of the compression graph for testing # endif

Finally, I have analyzed the redis source code for some time. I have synchronized the analyzed code to my personal github. Let's take a look at the following link:

Github: https://github.com/linyiqun/Redis-Code