Nginx Hash Table Structure ngx_hash_t

Overview

The basics of hash tables are described in the previous article, "Data structure-hash table". The hash table combines the characteristics of arrays and lists, making it easier to address, insert, and delete operations. The process of a hash table is to map the keyword to the corresponding hash table location by a hash function, that is, the location of the hash table where the corresponding hashes are located. However, when multiple keyword mappings occur in the same location, resulting in a conflict problem, the hash table uses two selectable methods: Zipper method and Open addressing method, in order to resolve the situation.

Nginx hash table using open addressing to solve the conflict problem, in order to deal with strings, Nginx also implemented a support wildcard operation of the relevant functions, the following on Nginx hash table in the source code analysis. Source file: src/core/ngx_hash.h/.c.

Hash table structure

NGX_HASH_ELT_T structure

The structure of the key elements in the hash table is ngx_hash_elt_t, and the hashtable element structure takes the key-value form, i.e. <key,value>. It is defined as follows:

/* The structure of the elements in the hash hash list, using the key values and their corresponding values <key,value>*/typedef struct {    void             *value;    /* point to User-defined data */    u_short           len;      /* The length of key value */    U_char            name[1];  /* Key value key for the first character, the array name name represents the key to the value of the first address */} ngx_hash_elt_t;

ngx_hash_t Structure

Hash table basic structure ngx_hash_t, its structure is defined as follows:

/* Basic hash list structure */typedef struct {    ngx_hash_elt_t  **buckets;  /* A bucket that points to the first storage element of the hash list */    ngx_uint_t        size;     /* The number of buckets for hash hash list */} ngx_hash_t;

The element structure diagram and the basic hash structure diagram are as follows:

NGX_HASH_INIT_T initialization structure

Hash initialization structure NGX_HASH_INIT_T,NGINX hash initialization structure is ngx_hash_init_t, used to encapsulate its related data as parameters passed to Ngx_hash_init (), which is defined as follows:

/* Initialize hash structure */typedef struct {    ngx_hash_t       *hash;         /* point to the basic hash structure to initialize */    ngx_hash_key_pt   key;          /* Hash function pointer */    ngx_uint_t        max_size;     /* The maximum number of bucket buckets in the hash table */    ngx_uint_t        bucket_size;  /* Storage space per bucket of buckets */    char             *name;         /* The name of the hash structure (used only in the error log) */    ngx_pool_t       *pool;         /* The memory pool that allocates the hash structure *////////////    * Allocates temporary data space of memory pool, only before initializing hash table, allocates some temporary array */    ngx_pool_t       *temp_pool;} ngx_hash_init_t ;

hash element data ngx_hash_key_t, the structure is mainly used to save the data to hash, that is, the key-value pair <key,value> In practice, it is common to store multiple key-value pairs in an array of ngx_hash_key_t structures. Passed as a parameter to Ngx_hash_init (). It is defined as follows:

/* Computes the hash element structure of the element to be added */typedef struct {    ngx_str_t         key;      /* Element keyword */    ngx_uint_t        key_hash;/* Element keyword key calculates the hash value */    void             *value;    /* points to the keyword key corresponding to the value, composed of hash table elements: Key-value <key,value> */} ngx_hash_key_t;

Hashing operations

Hashing operations include initialization functions, lookup functions, where the initialization function is the Nginx table is more important function, because nginx hash table is static read-only, that is, can not dynamically add new elements at run time, all the structure and data are configured at the time of initialization has been planned.

hash function

The hash table uses a hash function to map the user data to the corresponding location in the Hashtable, following the definition of the Nginx hash function:

/* Hash function */#define NGX_HASH (Key, C) ((ngx_uint_t) key * + c) ngx_uint_t Ngx_hash_key (U_char *data, size_t len); ngx_ui nt_t NGX_HASH_KEY_LC (U_char *data, size_t len), ngx_uint_t Ngx_hash_strlow (U_char *dst, U_char *src, size_t N); #define Ngx  _hash (Key, C) ((ngx_uint_t) key * + C)/* hash function */ngx_uint_tngx_hash_key (U_char *data, size_t len) {ngx_uint_t I,    Key    Key = 0;    for (i = 0; i < len; i++) {/* Call the hash function defined by the macro */key = Ngx_hash (key, Data[i]); } return key;    /* Here just convert all the characters in the string data to lowercase letters and then hash value */NGX_UINT_TNGX_HASH_KEY_LC (U_char *data, size_t len) {ngx_uint_t I, key;    Key = 0;    for (i = 0; i < len; i++) {/* Converts a string to lowercase characters and calculates the hash value of each character */key = Ngx_hash (key, Ngx_tolower (Data[i])); } return key;    /* Convert the first n characters of the original key string to lowercase and then calculate the hash value * Note: The hash value of the first n characters is calculated here */ngx_uint_tngx_hash_strlow (U_char *dst, U_char *src, size_t N) {    ngx_uint_t key;    Key = 0; while (n--) {/* Converts the first n characters of the SRC string to lowercase letters */*DST = Ngx_tolower (*SRC);        Key = Ngx_hash (key, *DST);/* Calculates the hash value of the converted lowercase characters */dst++;    src++; Return key;/* returns the hash value of the integer */}

Hash initialization function

Hash initialization is done by the Ngx_hash_init () function, whose names parameter is an array of ngx_hash_key_t structures, that is, a key-value pair <key,value> array, and Nelts represents the number of elements in the array. The result of the function initialization is to save the names array of key-value pairs <key,value>, by hashing it into the corresponding one or more hash barrels (that is, buckets in the code). The hash bucket contains a pointer to the NGX_HASH_ELT_T structure (the hash element pointer), which points to a basic contiguous data area. The data area is stored in the hash after the key-value pair <key ', "value", that is, the ngx_hash_elt_t structure of the field <name,value>. Each such data area holds the key-value pair <key ', and value ' > can be one or more. It is defined as follows:

#define NGX_HASH_ELT_SIZE (name) (sizeof (void *) + ngx_align (name)->k Ey.len + 2, sizeof (void *)))/* Initialize the hash structure function *//* parameter hinit is the hash table initialization structure pointer; * Name is an array of elements to be added to the hash table structure; * Nelts is the number of elements in the array of elements to be added;    Ngx_int_tngx_hash_init (ngx_hash_init_t *hinit, ngx_hash_key_t *names, ngx_uint_t nelts) {U_char *elts;    size_t Len;    U_short *test;    ngx_uint_t I, N, key, size, start, bucket_size;    ngx_hash_elt_t *elt, **buckets; for (n = 0; n < nelts; n++) {/* If there is not enough memory space per bucket bucket to store a keyword element, an error is returned * This takes into account the space required for the last null pointer of each bucket, that is, in the statement sizeof (void *), * this pointer can be used as the closing tag in the lookup process */if (Hinit->bucket_size < Ngx_hash_elt_size (&names[                          N]) + sizeof (void *)) {Ngx_log_error (Ngx_log_emerg, Hinit->pool->log, 0, "Could not build the%s, you should" "Increase%s_bucket_size:%i", HI Nit-&gT;name, Hinit->name, hinit->bucket_size);        return ngx_error;  }}/* temporarily allocates sizeof (u_short) *max_size test space, that is, the test array has a total of max_size elements, that is, the number of maximum buckets, * each element will accumulate to the corresponding hash table position of the keyword length, * When greater than 256 bytes, which is the byte size represented by U_short, * indicates less buckets */test = Ngx_alloc (hinit->max_size * sizeof (u_short), hinit->p    Ool->log);    if (test = = NULL) {return ngx_error; }/* The size of the data that is actually accommodated per bucket, * because the end of each bucket is null, * so the size of the bucket that actually holds the data must be subtracted from the memory size of a pointer */bucket_size = h    init->bucket_size-sizeof (void *); /* Estimate the minimum number of buckets for the hash table; * The memory space required for each keyword element is ngx_hash_elt_size (&name[n]), at least two pointers are required (2*sizeof (void *)) * To estimate hash The minimum number of buckets required for the table * because the smaller the key element memory, the more key elements each bucket holds * the less the bucket of the hash table requires, but at least one bucket */start = Nelts    /(Bucket_size/(2 * sizeof (void *))); Start = start?    Start:1;  /* Estimate the maximum number of buckets required for the hash table, i.e. max_size */if (Hinit->max_size > 10000 && nelts && hinit->max_size/ NelTS < +) {start = hinit->max_size-1000; }/* To estimate the number of buckets required by the hash table to accommodate the Nelts key element by testing the array test start with the minimum number of buckets previously estimated * Increase the number of buckets as needed * * for (size = STA Rt Size <= hinit->max_size;        size++) {Ngx_memzero (test, size * sizeof (u_short));            for (n = 0; n < nelts; n++) {if (Names[n].key.data = = NULL) {continue;            */* Calculated from the hash value of the keyword element exists in the location of the test array testing corresponding to */key = names[n].key_hash% size; Test[key] = (u_short) (Test[key] + ngx_hash_elt_size (&names[n])); #if 0 ngx_log_error (Ngx_log_alert, hinit-& Gt;pool->log, 0, "%ui:%ui%ui \"%v\ "", size, key, Test[key], &na             Mes[n].key); #endif/* Test array with memory greater than bucket bucket maximum memory, you need to expand the number of buckets * that is, on the basis of start to continue to increase the value of size            */if (Test[key] > (u_short) bucket_size) {goto next; }}/* If size bucket canTo accommodate all the key elements of the name array, it means that the size of the bucket to find the appropriate number of buckets */goto found;    Next:continue; } ngx_log_error (Ngx_log_warn, Hinit->pool->log, 0, "could not build optimal%s, you should incre ASE "" Either%s_max_size:%i or%s_bucket_size:%i; "" Ignoring%s_bucket_size ", Hinit->name, Hinit->name, Hinit->max_size, hinit->    Name, Hinit->bucket_size, hinit->name); found:/* To this has found the appropriate bucket quantity, that is, the size * re-initialize the test array element, the initial value is a pointer size */    for (i = 0; i < size; i++) {Test[i] = sizeof (void *);        }/* Calculates the space occupied by the keywords in each bucket, that is, the size of the data that each bucket actually holds, * it must be noted that there is also a pointer size in test[i] */for (n = 0; n < nelts; n++) {        if (Names[n].key.data = = NULL) {continue;        }/* Calculates the keyword in the corresponding Test[key] according to the hash value, i.e. the size of Test[key] increases the size of a key element */key = names[n].key_hash% size;    Test[key] = (u_short) (Test[key] + ngx_hash_elt_size (&names[n])); }    len = 0;            /* Adjust the size to Cacheline and record the total length of all elements */for (i = 0; i < size; i++) {if (test[i] = = sizeof (void *)) {        Continue        } Test[i] = (u_short) (Ngx_align (Test[i], ngx_cacheline_size));    Len + = Test[i]; }/* * Request memory space for bucket elements in the memory pool, * Note: If the hash header structure is not previously requested, this will be applied with ngx_hash_wildcard_t */if (Hinit->hash =                                             = NULL) {Hinit->hash = Ngx_pcalloc (hinit->pool, sizeof (ngx_hash_wildcard_t)        + Size * sizeof (ngx_hash_elt_t *));            if (Hinit->hash = = NULL) {ngx_free (test);        return ngx_error; }/* Calculates the starting position of the buckets */buckets = (ngx_hash_elt_t *) ((U_char *) Hinit->hash + Sizeo    F (ngx_hash_wildcard_t));        } else {buckets = Ngx_pcalloc (hinit->pool, size * sizeof (ngx_hash_elt_t *));            if (buckets = = NULL) {ngx_free (test);        return ngx_error; }}/* Assign ELTs, align to CAcheline Size */ELTs = Ngx_palloc (hinit->pool, Len + ngx_cacheline_size);        if (ELTs = = NULL) {ngx_free (test);    return ngx_error;    } ELTs = Ngx_align_ptr (ELTs, ngx_cacheline_size);  /* Match the buckets array to the corresponding ELTs, that is, set the address of each bucket corresponding to the actual data */for (i = 0; i < size; i++) {if (test[i] = sizeof (void *))        {continue;        } Buckets[i] = (ngx_hash_elt_t *) ELTs;    ELTs + = Test[i];    }/* Empties the test array so that it accumulates the length of the actual data, and does not calculate the length of the trailing pointer */for (i = 0; i < size; i++) {test[i] = 0; }/* Fill in each bucket with the actual data */for (n = 0; n < nelts; n++) {if (Names[n].key.data = = NULL) {Conti        Nue        } key = names[n].key_hash% size;        ELT = (ngx_hash_elt_t *) ((U_char *) Buckets[key] + test[key]);        Elt->value = Names[n].value;        Elt->len = (u_short) Names[n].key.len;        Ngx_strlow (Elt->name, Names[n].key.data, Names[n].key.len);     /* Test[key] records the fill position of the current bucket content, that is, the starting position of the next fill */   Test[key] = (u_short) (Test[key] + ngx_hash_elt_size (&names[n]));        }/* Sets the NULL pointer for the bucket end position */for (i = 0; i < size; i++) {if (buckets[i] = = NULL) {continue;        } ELT = (ngx_hash_elt_t *) ((U_char *) buckets[i] + test[i]);    Elt->value = NULL;    } ngx_free (test);    Hinit->hash->buckets = buckets;        hinit->hash->size = size, #if 0 for (i = 0; i < size; i++) {ngx_str_t val;        ngx_uint_t key;        ELT = Buckets[i]; if (ELT = = NULL) {ngx_log_error (Ngx_log_alert, Hinit->pool->log, 0, "%ui:null"            , i);        Continue            } while (elt->value) {Val.len = elt->len;            Val.data = &elt->name[0];            Key = Hinit->key (Val.data, Val.len); Ngx_log_error (Ngx_log_alert, Hinit->pool->log, 0, "%ui:%p \"%v\ "%ui", I, ELT, &val, K            EY); ELT = (ngx_hash_elt_t *) ngx_align_ptr (&elt->name[0] + elt->len, sizeof (VO        ID *)); }} #endif return NGX_OK;}

Hash Lookup function

The hash lookup operation is done by the Ngx_hash_find () function, where the bucket is computed directly by key, where the starting address of the ngx_hash_elt_t data area is saved, and then the match is made based on the length and the name content, The value field of its ngx_hash_elt_t structure is the one that is being asked. It is defined as follows:

/* find hash Element */    void *ngx_hash_find (ngx_hash_t *hash, ngx_uint_t key, U_char *name, size_t len) {ngx_uint_t i; ngx_hash_elt_t *elt; #if 0 ngx_log_error (ngx_log_alert, Ngx_cycle->log, 0, "hf:\"%*s\ "", Len, name), #endif/* by Ke    Y find where the element is in the bucket in the hash table */ELT = hash->buckets[key% hash->size];    if (ELT = = null) {return null;        The while (Elt->value) {if (len! = (size_t) elt->len) {/* Determines whether the length is equal/goto next;                } for (i = 0; i < len; i++) {if (Name[i]! = Elt->name[i]) {/* If the length is equal, the contents of name are compared */            Goto Next;    }/* Match succeeded, return value field */Return elt->value;                                               Next:elt = (ngx_hash_elt_t *) ngx_align_ptr (&elt->name[0] + Elt->len,        sizeof (void *));    Continue } return NULL;} 

Resources:

"Deep understanding of Nginx"

Design and implementation of the hash table in Nginx

Nginx Hash Table Structure ngx_hash_t