1. why do we need dynamic hash common hash? Most of them are the following faces: a static hash structure maintains some buckets, that is, the left part of the graph, each bucket contains data with the same hash value. These data with the same hash value form a linked list. One of the main disadvantages of this hash is that the number of buckets is fixed.
1. why do we need dynamic hash common hash? Most of them are the following faces: a static hash structure maintains some buckets, that is, the left part of the graph, each bucket contains data with the same hash value. These data with the same hash value form a linked list. One of the main disadvantages of this hash is that the number of buckets is fixed.
1. Why dynamic hash?
The common hash is mostly the following face:
A static hash structure
This Hash maintains some buckets, that is, the left part of the graph. Each bucket contains data with the same hash value.
These data with the same hash value form a linked list. One of the main disadvantages of this hash method is that the number of buckets is fixed and not easy to expand. As the number of inserted data increases, the search efficiency decreases sharply.
Dynamic hash is used to solve this problem. The dynamic hash implemented by postgresql ensures that the size of the hash table is dynamically increased when the fill factor does not exceed a predetermined value. At the same time, the changes made to each expansion are not significant, and the space utilization is also relatively high.
2. Dynamic hash structure
The Postgresql Code related to dynamic hash is distributed in the files dynahash. c and hashfn. c. hashfn. c.
It mainly involves some Hash functions, while dynahash. c is the main implementation of dynamic hash.
Compared with normal hash tables, dynamic hash has a new administrative unit: directory. For example:
Postgresql dynamic hash structure
Dir is a variable-size array. The initial length can be specified during creation. In the future, the length of each extension will be X2. Each item in dir points to a fixed Segment with the same length and must be an integer power of 2. The element in the Segment array is a Bucket ), each bucket stores a linked list. Dynamic hash stores all elements with the same hash value in the same bucket.
Now let's take a look at the definitions of these basic concepts in pg:
Typedef struct HASHELEMENT
{
Struct HASHELEMENT * link;/* link to next entry in same bucket */
Uint32 hashvalue;/* hash function result for this entry */
} HASHELEMENT;
/* A hash bucket is a linked list of HASHELEMENTs */
Typedef HASHELEMENT * HASHBUCKET;
/* A hash segment is an array of bucket headers */
Typedef HASHBUCKET * HASHSEGMENT;
All these definitions can correspond to each other.
3. How to Find the Bucket corresponding to the given hash value
Let's take a look at the implementation:
/* Convert a hash value to a bucket number */
Static inline uint32
Calc_bucket (HASHHDR * hctl, uint32 hash_val)
{
Uint32 bucket;
Bucket = hash_val & hctl-> high_mask;
If (bucket> hctl-> max_bucket)
Bucket = bucket & hctl-> low_mask;
Return bucket;
}
Hctl-> max_bucket indicates that the total number of buckets is reduced by 1. For example, this value is 15.
Hctl-> low_mask is the maximum 2 ^ K minus 1 of <= (hctl-> max_bucket + 1). For example, the value is 16-1 = 15 (0000 1111)
Hctl-> high_mask is 2 ^ (K + 1) minus 1. For example, this value is 32-1 = 31 (0x0001 1111)
Note that the hctl-> max_bucket changes after the hash table is created. Generally, one is added at a time, if the hctl-> max_bucket becomes the integer power of 2, you need to update hctl-> low_mask and hctl-> high_mask. The update code is as follows:
/*
* If we crossed a power of 2, readjust masks.
*/
If (uint32) new_bucket> hctl-> high_mask)
{
Hctl-> low_mask = hctl-> high_mask;
Hctl-> high_mask = (uint32) new_bucket | hctl-> low_mask;
}
Let's look back at the calc_bucket function, because understanding this function is the key to understanding dynamic expansion. From the introduction of max_bucket, low_mask and high_mask above, we can draw the following conclusion:
Hctl-> high_mask> = hctl-> max_bucket> = hctl-> low_mask
In their binary representation, if hctl-> low_mask occupies m (2 ^ m-1), hctl-> high_mask occupies m + 1. The hctl-> max_bucket is between the closed range [low_mask, high_mask. Therefore, to obtain the bucket value of hash_val, the priority should be "higher than high_mask". If the obtained bucket number is larger than that of max_bucket, the "upper low_mask" parameter should be obtained. This step laid the groundwork for subsequent extensions. This problem may occur when max_bucket is added during expansion. The values of the same hash_val before and after expansion are different through calc_bucket. The following describes the extension solution in detail.
4. Dynamic Scaling
After figuring out the structure of dynamic hash and how to make it worthwhile from hash to its bucket, it is very easy to search, delete, and insert hash tables. For example, to delete a bucket, call cal_bucket to find its bucket, and then find all the elements with the same key value in the bucket. Then, delete the elements from the linked list corresponding to the bucket. The following describes the dynamic expansion problem. After all, this is the core of pg dynamic hash.
When it comes to dynamic expansion, we have to talk about the expansion time. When do we need to increase the capacity of our hash table? We increase the capacity to ensure the search efficiency. The search efficiency of the hash table is closely related to a fill factor. The filling factor of pg is defined:
Fill Factor = hctl-> nentries/(hctl-> max_bucket + 1)
From this formula, we can see that the fill factor will increase as the number of inserted elements increases. When it increases to a greater value than the hctl-> ffactor, it needs to be extended, the extension here means that one more hctl-> max_bucket is required. The procedure is as follows:
1) Calculate the segment index value of max_bucket + 1.
< <扩展dir和segment> >
2) If segndx does not exceed the capacity of the allocated segment, move to 5); otherwise, continue;
3) Check whether segndx exceeds the existing dir size. If yes, extend the dir size to twice the previous size;
4) assign a new segment to dir [segndx;
< <计算新旧bucket> >
5) Calculate the max_bucket + 1 bucket Number old_bucket before expansion;
6) After max_bucket ++, check whether max_bucket has the integer power of 2. If so, modify low_mask and high_mask;
7) Calculate the new bucket number new_bucket;
< <将old_bucket中满足条件的元素移动new_bucket中> >
8) traverse the linked list of the old_bucket, recalculate the bucket they should be in, and move the bucket to new_bucket.
In this process, only step 8 needs to be described. Why is the old_bucket of the previous expansion not checked? What about the old_bucket of the previous expansion? This is determined by the bucket Calculation Method in calc_bucket. In fact, what you think is correct is that you should traverse the old_bucket extended once or twice, but it is not necessary, even if you do this, you cannot find an element from these buckets that can be imported into new_bucket. We only need to prove that after one extension, the bucket corresponding to the hash value of the elements left in the old_bucket will not change any more.
Before and after expansion, the bucket computing path can be expressed using the following matrix:
After & high_mask extension & low_mask
Before Expansion & high_mask (1) (2)
Before Expansion & low_mask (3) (4)
In this case, (2) is not possible because the high_mask before the extension is equal to the low_mask after the extension. If (2) is set up, there is a conflict between max_bucket> (max_bucket + 1. In the case of (1), because the high_mask and max_bucket will only increase, the bucket numbers will not change, and (3) and (4) it is actually the item to be moved for this extension. So it has been proved that the elements that remain in the old_bucket will not change.
5. Summary
The dynamic hash implemented by pg ensures that the fill factor does not exceed the set value, and its retrieval efficiency will not be reduced because of the increase in inserted elements. At the same time, the expansion cost is not very high, you only need to move some items in a bucket to the new bucket at a time.