Redis storage key-Value Method

Source: Internet
Author: User
Tags rehash redis server

Redis storage key-Value Method

Redis is an in-memory database that stores key-value pairs. Its key-value storage method is similar to HashMap in Java. The structure used to characterize the redis database is redisDb (in the server. h file). By default, the redis server has 16 databases numbered from 0 to 15.

Typedef struct redisDb {
Dict * dict;/* key space */
Dict * expires;/* expired key space */
Dict * blocking_keys;/* Key waiting by the client (BLPOP )*/
Dict * ready_keys;/* receives the push blocking key */
Dict * watched_keys;/* WATCHED keys for MULTI/exec cas */
Struct evictionPoolEntry * eviction_pool;/* Eviction pool of keys */
Int id;/* Database ID */
Long avg_ttl;/* Average TTL, just for stats */
} RedisDb;

In dict, key-> value is stored, and expires stores key-> expiration time.

Dict is the struct defined in dict. h:

Typedef struct dict {
DictType * type;
Void * privdata;
Dictht ht [2];
Long rehashidx;/* rehashing not in progress if rehashidx =-1 */
Unsigned long iterators;/* number of iterators currently running */
} Dict;

Typedef struct dictht {
DictEntry ** table;
Unsigned long size; // table size
Unsigned long sizemask;
Unsigned long used; // number of key-value pairs in the table
} Dictht;

Typedef struct dictEntry {
Void * key;
Union {
Void * val;
Uint64_t u64;
Int64_t s64;
Double d;
} V;
Struct dictEntry * next;
} DictEntry;

Dict can be analogous to HashMap in java. dictEntry corresponds to java. util. hashMap. entry <K, V>, slightly different: dict encapsulates the entry table (dictht), and there are two tables in dict for rehash.

Analyze dict's dictReplace (in the dict. c file), similar to HashMap's put:

/* Add or Overwrite:
* Add an element, discarding the old value if the key already exists.
* Return 1 if the key was added from scratch, 0 if there was already
* Element with such key and dictReplace () just synchronized med a value update
* Operation .*/
Int dictReplace (dict * d, void * key, void * val)
{
DictEntry * entry, * existing, auxentry;

/* Try to add the element. If the key
* Does not exists dictAdd will suceed .*/
Entry = dictAddRaw (d, key, & existing );
If (entry ){
DictSetVal (d, entry, val );
Return 1;
}

/* Set the new value and free the old one. Note that it is important
* To do that in this order, as the value may just be exactly the same
* As the previous one. In this context, think to reference counting,
* You want to increment (set), and then decrement (free), and not
* Reverse .*/
Auxentry = * existing;
DictSetVal (d, existing, val );
DictFreeVal (d, & auxentry );
Return 0;
}

DictEntry * dictAddRaw (dict * d, void * key, dictEntry ** existing)
{
Int index;
DictEntry * entry;
Dictht * ht;

If (dictIsRehashing (d) _ dictRehashStep (d );

/* Get the index of the new element, or-1 if
* The element already exists .*/
If (index = _ dictKeyIndex (d, key, dictHashKey (d, key), existing) =-1)
Return NULL;

/* Allocate the memory and store the new entry.
* Insert the element in top, with the assumption that in a database
* System it is more likely that recently added entries are accessed
* More frequently .*/
Ht = dictIsRehashing (d )? & D-> ht [1]: & d-> ht [0];
Entry = zmalloc (sizeof (* entry ));
Entry-> next = ht-> table [index];
Ht-> table [index] = entry;
Ht-> used ++;

/* Set the hash entry fields .*/
DictSetKey (d, entry, key );
Return entry;
}

In dictAddRaw, the main logic is to first obtain the index in the table, and then insert it into the table's linked list using the header insertion method.

If dict is in rehash state (that is, rehashidx! =-1), The _ dictRehashStep is called at the time of insertion. For dict in rehash, the table used is ht [1].

Static void _ dictRehashStep (dict * d ){
If (d-> iterators = 0) dictRehash (d, 1 );
}

Int dictRehash (dict * d, int n ){
Int empty_visits = n * 10;/* Max number of empty buckets to visit .*/
If (! DictIsRehashing (d) return 0;

While (n -- & d-> ht [0]. used! = 0 ){
DictEntry * de, * nextde;

/* Note that rehashidx can't overflow as we are sure there are more
* Elements because ht [0]. used! = 0 */
Assert (d-> ht [0]. size> (unsigned long) d-> rehashidx );
While (d-> ht [0]. table [d-> rehashidx] = NULL ){
D-> rehashidx ++;
If (-- empty_visits = 0) return 1;
}
De = d-> ht [0]. table [d-> rehashidx];
/* Move all the keys in this bucket from the old to the new hash HT */
While (de ){
Unsigned int h;

Nextde = de-> next;
/* Get the index in the new hash table */
H = dictHashKey (d, de-> key) & d-> ht [1]. sizemask;
De-> next = d-> ht [1]. table [h];
D-> ht [1]. table [h] = de;
D-> ht [0]. used --;
D-> ht [1]. used ++;
De = nextde;
}
D-> ht [0]. table [d-> rehashidx] = NULL;
D-> rehashidx ++;
}

/* Check if we already rehashed the whole table ...*/
If (d-> ht [0]. used = 0 ){
Zfree (d-> ht [0]. table );
D-> ht [0] = d-> ht [1];
_ DictReset (& d-> ht [1]);
D-> rehashidx =-1;
Return 0;
}

/* More to rehash ...*/
Return 1;
}

The Code shows that rehashidx marks the index of the rehash linked list in ht [0.

So under what circumstances will redis rehash dict?

Call Stack: _ dictKeyIndex-> _ dictExpandIfNeeded-> dictExpand. When calculating the index of the key, the system determines whether dict needs to be extended. If it needs to be extended, the rehashidx of dict is set to 0.


Static int _ dictKeyIndex (dict * d, const void * key, unsigned int hash, dictEntry ** existing)
{
Unsigned int idx, table;
DictEntry * he;
If (existing) * existing = NULL;

/* Expand the hash table if needed */
If (_ dictExpandIfNeeded (d) = DICT_ERR)
Return-1;
For (table = 0; table <= 1; table ++ ){
Idx = hash & d-> ht [table]. sizemask;
/* Search if this slot does not already contain the given key */
He = d-> ht [table]. table [idx];
While (he ){
If (key = he-> key | dictCompareKeys (d, key, he-> key )){
If (existing) * existing = he;
Return-1;
}
He = he-> next;
}
If (! DictIsRehashing (d) break;
}
Return idx;
}

/* Expand the hash table if needed */
Static int _ dictExpandIfNeeded (dict * d)
{
/* Incremental rehashing already in progress. Return .*/
If (dictIsRehashing (d) return DICT_ OK;

/* If the hash table is empty expand it to the initial size .*/
If (d-> ht [0]. size = 0) return dictExpand (d, DICT_HT_INITIAL_SIZE );

/* If we reached the 1:1 ratio, and we are allowed to resize the hash
* Table (global setting) or we shoshould avoid it but the ratio
* Elements/buckets is over the "safe" threshold, we resize doubling
* The number of buckets .*/
If (d-> ht [0]. used> = d-> ht [0]. size &&
(Dict_can_resize |
D-> ht [0]. used/d-> ht [0]. size> dict_force_resize_ratio ))
{
Return dictExpand (d, d-> ht [0]. used * 2 );
}
Return DICT_ OK;
}

/* Expand or create the hash table */
Int dictExpand (dict * d, unsigned long size)
{
Dictht n;/* the new hash table */
Unsigned long realsize = _ dictNextPower (size );

/* The size is invalid if it is smaller than the number
* Elements already inside the hash table */
If (dictIsRehashing (d) | d-> ht [0]. used> size)
Return DICT_ERR;

/* Rehashing to the same table size is not useful .*/
If (realsize = d-> ht [0]. size) return DICT_ERR;

/* Allocate the new hash table and initialize all pointers to NULL */
N. size = realsize;
N. sizemask = realsize-1;
N. table = zcalloc (realsize * sizeof (dictEntry *));
N. used = 0;

/* Is this the first initialization? If so it's not really a rehashing
* We just set the first hash table so that it can accept keys .*/
If (d-> ht [0]. table = NULL ){
D-> ht [0] = n;
Return DICT_ OK;
}

/* Prepare a second hash table for incremental rehashing */
D-> ht [1] = n;
D-> rehashidx = 0;
Return DICT_ OK;
}

From the data structure point of view, redis's dict and java's HashMap are very similar, the difference is that rehash: HashMap is a one-time copy at resize, and then use the new array, dict maintains two dictht, and usually uses ht [0]. Once rehash is started, ht [0] and ht [1] are used. rehash is distributed to operations such as dictAdd and dictFind each time.

DictEntry * dictFind (dict * d, const void * key)
{
DictEntry * he;
Unsigned int h, idx, table;

If (d-> ht [0]. used + d-> ht [1]. used = 0) return NULL;/* dict is empty */
If (dictIsRehashing (d) _ dictRehashStep (d );
H = dictHashKey (d, key );
For (table = 0; table <= 1; table ++) {// traverses d-> ht [0] and d-> ht [1]
Idx = h & d-> ht [table]. sizemask;
He = d-> ht [table]. table [idx];
While (he ){
If (key = he-> key | dictCompareKeys (d, key, he-> key ))
Return he; // return if it is found
He = he-> next;
}
If (! DictIsRehashing (d) return NULL;
}
Return NULL;
}

Why is redis designed like this?

Imagine if redis is also a one-time copy like HashMap in java, when the dict is very large, the copy will be time-consuming. During this period, redis cannot provide external services.

This design adds complexity. After rehash is started, dict data is scattered in ht [0] and ht [1], for query (dictFind) and delete (dictDelete) and (dictReplace), the ht [0] and ht [1] will be traversed.

This article permanently updates link: https://www.bkjia.com/Linux/2018-03/151173.htm

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.