Dict Dictionary of Redis underlying data structures 1

Recently, I wanted to learn Redis by using Redis's source code. Although the usual work is not used much, but the Redis is still more interested in, after all, its performance is good. Redis is an open source project that we can use to understand Redis through source code. I will later through their own study, write some about the Redis source code posts. The main content of the post is the analysis of the code design, and does not explain the source code in detail. If there is a wrong place, please correct me. Source code is Reids 3.0.3 version.

Dict Dictionary

First, the data structure

// dictionary entry
typedef struct dictEntry {
    void * key;
    union {
        void * val;
        uint64_t u64;
        int64_t s64;
        double d;
    } v;
    struct dictEntry * next;
} dictEntry;
typedef struct dictType {
    unsigned int (* hashFunction) (const void * key); // Calculate the hash of the key
    void * (* keyDup) (void * privdata, const void * key); // function to copy key
    void * (* valDup) (void * privdata, const void * obj); // function to copy value
    int (* keyCompare) (void * privdata, const void * key1, const void * key2); // compare functions with equal keys
    void (* keyDestructor) (void * privdata, void * key); // function to destroy key
    void (* valDestructor) (void * privdata, void * obj); // function to destroy value
} dictType;
// Hash table for storing dictionary conditions
/ * This is our hash table structure. Every dictionary has two of this as we
 * implement incremental rehashing, for the old to the new table. * /
typedef struct dictht {
    dictEntry ** table;
    unsigned long size;
    unsigned long sizemask;
    unsigned long used;
} dictht;
//dictionary
typedef struct dict {
    dictType * type;
    void * privdata;
    dictht ht [2]; // Two hash tables, two hash tables will be used when rehash, otherwise only one table will be used
    long rehashidx; / * rehashing not in progress if rehashidx == -1 * /
    int iterators; / * number of iterators currently running * /
dict;
/ * If safe is set to 1 this is a safe iterator, that means, you can call
 * dictAdd, dictFind, and other functions against the dictionary even while
 * iterating. Otherwise it is a non safe iterator, and only dictNext ()
 * should be called while iterating. * /
typedef struct dictIterator {
    dict * d;
    long index;
    int table, safe;
    dictEntry * entry, * nextEntry;
    / * unsafe iterator fingerprint for misuse detection. * /
    long long fingerprint;
} dictIterator;

Ii. simple functions of macro implementation

Give three examples:

#define dictFreeVal(d, entry)     if ((d)->type->valDestructor)         (d)->type->valDestructor((d)->privdata, (entry)->v.val)
#define dictSetVal(d, entry, _val_) do {     if ((d)->type->valDup)         entry->v.val = (d)->type->valDup((d)->privdata, _val_);     else         entry->v.val = (_val_); } while(0)
#define dictSetSignedIntegerVal(entry, _val_)     do { entry->v.s64 = _val_; } while(0)

Dictfreeval, which is used when releasing the value of a dictionary entry. The implementation did not use Do{}while (0), I did not want to understand why not to use, but I think it should be added, otherwise improper use will be problematic, specifically, I have another post: http://chhquan.blog.51cto.com/1346841/1358254

Dictsetsignedintegerval plus do{}while (0) should be used to prevent the use of macros in the form of expressions.


Third, part of the Code parsing

Because dict behavior characteristic is more, this post intends to explain part of the code.

1. Dict_can_resize

/* Using dictEnableResize() / dictDisableResize() we make possible to
 * enable/disable resizing of the hash table as needed. This is very important
 * for Redis, as we use copy-on-write and don‘t want to move too much memory
 * around when there is a child performing saving operations.
 *
 * Note that even when dict_can_resize is set to 0, not all resizes are
 * prevented: a hash table is still allowed to grow if the ratio between
 * the number of elements and the buckets > dict_force_resize_ratio. */
static int dict_can_resize = 1;
static unsigned int dict_force_resize_ratio = 5;

Dict_can_resize, which controls whether Dict can be rehash,1 when rehash,0 is allowed-generally not allowed rehash, but if the number of entries/buckets > Dict_force_resize_ratio is met, Rehash can still be performed. The dict_can_resize can be set by Dictenableresize () or dictdisableresize (). The purpose of this setting is to: when Redis needs to save the Dict (write file), is to save the current snapshot of dict, to keep the dict unchanged, but this will make the dictionary can not receive write operations or rehash, in order to ensure that the dict can handle the request normally, Redis adopts the Copy-on-write strategy, that is, when Dict has a modification operation, it is necessary to copy the Dict to support both the save operation and the modification operation. Because rehash also modifies dict and may cause the dict being saved to replicate, using dict_force_resize_ratio prevents rehash to some extent from replication. However, if the number of entries/buckets > Dict_force_resize_ratio is saved, Redis thinks that at this point the number of entries in Dict is too much relative to the bucket, and some buckets may have more elements hanging on them, which can have a serious effect on dict efficiency. So at this point rather copy dict also allow rehash to restore dict performance. Of course the specific dict_force_resize_ratio is how much, should be obtained by experiment. Or how to measure replication and maintain dict efficient turning points are also to be experimented with, not necessarily the number of entries/barrels, specifically by the experiment. As there is no experiment, I can't say more.

2. Hash calculation

Calculate the hash value of the function, the specific algorithm I am not familiar with, skip.

3. Reset Hash Table

// Reset the hash table
/ * Reset a hash table already initialized with ht_init ().
  * NOTE: This function should only be called by ht_destroy (). * /
static void _dictReset (dictht * ht)
{
     // The table value is directly overwritten below. The caller must ensure that the table does not point to a piece of dynamic memory.
     // Either dynamic memory has been released, or there are other pointers to reserve the dynamic memory space pointed to by table
     ht-> table = NULL;
     ht-> size = 0;
     ht-> sizemask = 0;
     ht-> used = 0;
}

4. Create Dict

// create dict
/ * Create a new hash table * /
dict * dictCreate (dictType * type,
         void * privDataPtr)
{
     dict * d = zmalloc (sizeof (* d));
     _dictInit (d, type, privDataPtr);
     return d;
}

5. Initialize Dict

// initialize dict
/ * Initialize the hash table * /
int _dictInit (dict * d, dictType * type,
         void * privDataPtr)
{
     _dictReset (& d-> ht [0]);
     _dictReset (& d-> ht [1]);
     d-> type = type;
     d-> privdata = privDataPtr;
     d-> rehashidx = -1; // -1 is not in rehash state, and> = 0 is in rehash
     d-> iterators = 0;
     return DICT_OK;
}

6. Resizing

// resize, resize according to the number of entries stored in the dict, the scalable hash table space can also be reduced.
/ * Resize the table to the minimal size that contains all the elements,
 * but with the invariant of a USED / BUCKETS ratio near to <= 1 * /
int dictResize (dict * d)
{
    int minimal;
    if (! dict_can_resize || dictIsRehashing (d)) return DICT_ERR;
    minimal = d-> ht [0] .used; // Resize by the number of entries stored
    if (minimal <DICT_HT_INITIAL_SIZE) // Minimum resize size is DICT_HT_INITIAL_SIZE
        minimal = DICT_HT_INITIAL_SIZE;
    return dictExpand (d, minimal);
}
/ * Expand or create the hash table * /
int dictExpand (dict * d, unsigned long size)
{
    dictht n; / * the new hash table * /
    unsigned long realsize = _dictNextPower (size); // Take the smallest power greater than 2 as the actual size
    / * the size is invalid if it is smaller than the number of
     * elements already inside the hash table * /
    if (dictIsRehashing (d) || d-> ht [0] .used> size)
        return DICT_ERR;
    / * Rehashing to the same table size is not useful. * /
    if (realsize == d-> ht [0] .size) return DICT_ERR;
    / * Allocate the new hash table and initialize all pointers to NULL * /
    n.size = realsize;
    n.sizemask = realsize-1; // size-1, bits with a value of 1 are all in the low order, used to take the size of the hash value as the bucket number of the hash table
    n.table = zcalloc (realsize * sizeof (dictEntry *));
    n.used = 0;
    / * Is this the first initialization? If so it ’s not really a rehashing
     * we just set the first hash table so that it can accept keys. * /
    if (d-> ht [0] .table == NULL) {
        d-> ht [0] = n;
        return DICT_OK;
    }
    / * Prepare a second hash table for incremental rehashing * /
    d-> ht [1] = n;
    d-> rehashidx = 0; //> = 0, in rehash
    return DICT_OK;
}

7. Rehash

// rehash function, rehash the dict step by step
// redis does not complete the rehash of dict at one time, but divides the entire rehash process into many small rehash operations to complete,
// Each rehash will process up to a certain number of buckets, specified by the parameter n. Since some buckets are empty, to prevent rehash from being accessed all the time
// To the empty bucket makes the rehash process take too much time. The function sets a maximum of n * 10 buckets.
// redis In order to maintain stable performance, some operations that have a chance to take more time will be divided into smaller operations. Rehash is one example.
/ * Performs N steps of incremental rehashing. Returns 1 if there are still
 * keys to move from the old to the new hash table, otherwise 0 is returned.
 *
 * Note that a rehashing step consists in moving a bucket (that may have more
 * than one key as we use chaining) from the old to the new hash table, however
 * since part of the hash table may be composed of empty spaces, it is not
 * guaranteed that this function will rehash even a single bucket, since it
 * will visit at max N * 10 empty buckets in total, otherwise the amount of
 * work it does would be unbound and the function may block for a long time. * /
int dictRehash (dict * d, int n) {
    int empty_visits = n * 10; / * Max number of empty buckets to visit. * /
    if (! dictIsRehashing (d)) return 0;
    // Access the bucket in ht [0]. If the bucket is not empty, put the elements in the bucket into ht [1].
    while (n-- && d-> ht [0] .used! = 0) {
        dictEntry * de, * nextde;
        / * Note that rehashidx ca n‘t overflow as we are sure there are more
         * elements because ht [0] .used! = 0 * /
    // From rehashidx, rehashidx is a variable used to record the state of the rehash process
        assert (d-> ht [0] .size> (unsigned long) d-> rehashidx);
    // Find a non-empty bucket, the total number of visits is limited by empty_visits
        while (d-> ht [0] .table [d-> rehashidx] == NULL) {
            d-> rehashidx ++;
            if (--empty_visits == 0) return 1; // Return 1 means that the rehash has not been completed and needs to be performed
        }
        de = d-> ht [0] .table [d-> rehashidx];
    // Move all items in the bucket to ht [1]
        / * Move all the keys in this bucket from the old to the new hash HT * /
        while (de) {
            unsigned int h;
            nextde = de-> next;
            / * Get the index in the new hash table * /
            h = dictHashKey (d, de-> key) & d-> ht [1] .sizemask; // for bucket number
            de-> next = d-> ht [1] .table [h];
            d-> ht [1] .table [h] = de;
            d-> ht [0] .used--;
            d-> ht [1] .used ++;
            de = nextde;
        }
        d-> ht [0] .table [d-> rehashidx] = NULL;
        d-> rehashidx ++; // Rehashidx bucket has been processed, the next bucket
    }
    // If ht [0] has no more entries, you can switch ht [1] to ht [0] and reset ht [1].
    / * Check if we already rehashed the whole table ... * /
    if (d-> ht [0] .used == 0) {
        zfree (d-> ht [0] .table); // Free bucket space of ht [0]
        d-> ht [0] = d-> ht [1];
        _dictReset (& d-> ht [1]);
        d-> rehashidx = -1;
        return 0;
    }
    / * More to rehash ... * /
    return 1;
} 

As can be seen from the rehash process, ht[0] and ht[1] have entries in the rehash process, i.e. all entries in the dictionary are distributed in Ht[0] and ht[1],
And then the trouble came out. The main problems are as follows: (now how to solve the problem)
1. How to find key.
2. How to insert a new key.
3. How to delete a key.
4. How to traverse dict all entries, how to ensure the traversal order.
5. How to ensure that the rehash process constantly inserts, deletes entries, and rehash no errors.
6. How to ensure that the iterator is valid and correct.

This article is from the "Chhquan" blog, make sure to keep this source http://chhquan.blog.51cto.com/1346841/1788910

Dict Dictionary of Redis underlying data structures 1