Resize a hash table (reprinted)

Hash table)

 

The hash table actually consists of a bucket of size to form a bucket array table [0... size-1]. After an object is hashed, a corresponding value is obtained, so we put this object in the bucket table [value. When a bucket contains multiple objects, we organize the objects in the bucket into a linked list. This is called the zipper method in conflict processing.

 

Load Factor)

 

Assume that the number of buckets in a hash table is size and the number of stored elements is used. we call used/size the load factor loadfactor. generally, when loadfactor <= 1, the expected complexity of hash table search is O (1 ). therefore, each time an element is added to a hash table, we must ensure that it can be added only when loadfactor is <1.

 

Capacity Expansion (expand) & allocation Transfer

 

When we add a new element, once loadfactor is greater than or equal to 1, we cannot simply add elements to the hash table. After the addition, the loadfactor will be greater than 1, so that the expected time complexity of the search cannot be guaranteed to be constant. In this case, we should expand the capacity of the bucket array to increase the size. In this way, we can ensure that used/size is still less than or equal to 1 after adding elements, so as to ensure that the expected time complexity of the search is O (1). But how can we expand the capacity? The expansion of vector capacity in C ++ is a good method. As a result, we have the following idea: each time loadfactor = 1 is found in the hash table, a space of two times of the original bucket array (called the New bucket array) is created ), then, all the elements in the original bucket array are transferred to the new bucket array. Note that the transfer here requires elements to be re-hashed to the new bucket one by one. The reason will be discussed later.

The disadvantage of this method is that capacity expansion is completed once, and it takes a long time to transfer all elements in the hash table. In this way, when loadfactor = 1 in the hash table, it takes a long time to insert an element into it.
Dict in redis. the design idea in C is to use two hash tables for expansion and transfer: When loadfactor = 1 in the First hash table, if you want to insert an element into the dictionary, first, the capacity of the first hash table is doubled for the second hash table, and all the elements in a non-empty bucket of the first hash table are transferred to the second hash table, then, the elements to be inserted are stored in the second hash table. Insert the second element into the dictionary, transfer all the elements in a non-empty bucket of the first hash table to the second hash table, and store the elements in the second hash table ...... Until the first hash table is empty.

This policy allocates the transfer of all elements in the first hash table for multiple transfers, and the expected time complexity of each transfer is O (1 ). In this way, it will not take a long time to insert an element into the dictionary.

To better understand this process, let's look at the two structures in dict. h:

Typedef struct dictht {
Dictentry ** table;
Unsigned long size;
Unsigned long sizemask;
Unsigned long used;
} Dictht;

Typedef struct dict {
Dicttype * type;
Void * privdata;
Dictht HT [2];
Int rehashidx;/* rehashing not in progress if rehashidx =-1 */
Int iterators;/* Number of iterators currently running */
} Dict;

Dictht refers to the bucket array mentioned above. size is used to indicate the capacity, generally 2 ^ N, sizemask (generally 2 ^ n-1, binary represents N 1) used to modulo hash values. Used indicates the number of elements stored in the hash table.

Dict represents a dictionary consisting of two bucket arrays. type is a number of function pointers (hash functions and some processing functions of key and value ).

D-> rehashidx

 

The key to understanding this variable is:

D-> rehashidx indicates whether the new elements are stored in bucket array 0 or bucket array 1, it also specifies which bucket in D-> H [0] is transferred to D-> H [1.

When d-> rehashidx =-1, the newly added elements should be stored in the bucket array 0.

When d-> rehashidx! =-1 indicates that all the first non-empty bucket element in the bucket array 0 should be transferred to bucket array 1 (this process may be called Bucket transfer or rehash ). In this process, elements in non-empty buckets must be re-hashed to array 1 of the bucket, because d-> H [1]-> sizemask is different from D-> H [0]-> sizemask. At this time, the newly added elements should be stored in the bucket array 1, because the loadfactor of the bucket array 0 is 1, while the loadfactor of the bucket array 1 is less than 1.

 

When it is found that all the elements in the bucket array 0 are transferred to the bucket array 1, that is, the bucket array 0 is empty. Release the bucket array 0 space and point the pointer of the bucket array 0 to bucket array 1. Assign D-> rehashidx to-1, so that the bucket array 1 is empty. When the element is added next time, it is still added to the bucket array 0. Until the number of elements in the bucket array 0 exceeds the number of buckets, we can re-open the double space of the bucket array 0 to the bucket array 1, and modify D-> rehashidx = 0, in this way, the next element is added to the bucket array 1.

 

It is worth noting that, before each Delete, search, or replace operation, you can determine whether to perform bucket Transfer Based on the status of D-> rehashidx. This can speed up the transfer.

 

The following is a simplified pseudocode, which inserts the n elements of element [1. N] To dict in sequence to describe the process of capacity expansion and transfer:

// Initialize two hash tables
D-> H [0]. size = 4; D-> H [1]. Used = 0; // allocate four empty buckets
D-> H [1]. size = 0; D-> H [1]. Used = 0; // Initialize an empty table

For (I = 1; I <= N; ++ I ){
If (D-> rehashidx! =-1 ){
If (D-> H [0]-> used! = 0 ){
Transfers a non-empty bucket element in D-> H [0] (re-hash) to D-> H [1;
// The previous step will make:
// D-> H [0]-> used-= number of transferred Elements
// D-> H [1]-> used + = number of transferred elements;
Hash element [I] to D-> H [1]; // D-> H [1]-> used ++
} Else {
// Use the bucket array 1 to overwrite the bucket array 0. The space of D-> H [0] must be released before the value is assigned. Reset the value D-> H [1]).
D-> H [0] = D-> H [1];
D-> rehashidx =-1;
Hash element [I] to D-> H [0]; // D-> H [0]-> used ++;
}
} Else if (D-> H [0]-> used> = D-> H [0]-> size)
D-> H [1] = new bucket [2 * D-> H [0]-> size];
// D-> H [0]-> size equals D-> H [0]-> twice the size
Hash element [I] to D-> H [1]; // D-> H [1]-> used ++
D-> rehashidx = 0;
} Else {
Hash element [I] to D-> H [0]; // D-> H [0]-> used ++
}
}

Dictionary iterator)

 

DividedSecurity iterator(Safe iterator) andNon-secure iterator.

The security iterator ensures that no bucket transfer is performed between two hash tables in the dictionary before the iterator is released.

The Influence of Bucket transfer on the iterator is very large. Assume that an iterator points to an element entity in a bucket D-> H [0]. After a bucket transfer, this object is rehash to D-> H [1. In D-> H [1], I don't know which elements have been missed by the iterator and which have not been accessed, this may allow the iterator to repeatedly access or missing some elements in the access dictionary. Therefore, the security iterator can ensure that not many elements are accessed repeatedly (of course, the operation of inserting and deleting new elements cannot be involved in the iteration process ).

Resize a hash table (reprinted)