Nginx learns four memory pool ngx_pool_t and memory management operations

I 've been watching nginx over the past few days and found that pool exists in all the places where I have memory application. Let's take a closer look. The original pool type is ngx_pool_t, which is used by nginx for memory management, so I decided to look at his implementation.

1 nginx memory pool-related struct

Ngx_pool_t is defined in core/ngx_palloc.h ngx_palloc.c. Below are several major struct types.

Ngx_pool_data_t

Typedef struct {// data structure module of the memory pool u_char * last; // The end position of the current memory allocation, that is, the starting position of the next allocable memory u_char * end; // The end position of the memory pool ngx_pool_t * Next; // link to the next memory pool. Many memory blocks in the memory pool are the ngx_uint_t failed linked by the pointer; // record the number of failures that the memory allocation cannot meet the requirement} ngx_pool_data_t; // The structure is used to maintain the data block of the memory pool for user allocation.

Ngx_pool_t

Struct ngx_pool_t {// memory pool management Allocation Module ngx_pool_data_t D; // data block size of the memory pool size_t Max; // data block size, maximum value of memory in small blocks ngx_pool_t * current; // point to the currently allocable memory pool ngx_chain_t * chain; // This pointer is attached to a ngx_chain_t structure ngx_pool_large_t * large; // points to the large memory allocation in nginx, the standard system interface malloc ngx_pool_cleanup_t * cleanup is directly used for memory allocation. // destructor. Some necessary operations for resources to be cleared when the memory is released are mounted. ngx_log_t * log; // log records related to memory allocation };

Other related struct

Typedef void (* ngx_pool_cleanup_pt) (void * data); typedef struct ngx_pool_cleanup_s handler; struct handler {ngx_pool_cleanup_pt handler; // clean function void * data; // memory ngx_pool_cleanup_t * Next; // run the clean function}; typedef struct into memory; struct ngx_pool_large_s {ngx_pool_large_t * Next; void * alloc; // large memory }; typedef struct {ngx_fd_t FD; u_char * Name; ngx_log_t * log;} ngx_pool_cleanup_file_t; // destroy the File Memory

2. functions related to memory pool operations

The following are functions related to nginx memory operations:

Void * ngx_alloc (size_t size, ngx_log_t * log); // malloc encapsulation void * ngx_calloc (size_t size, ngx_log_t * log); // malloc encapsulation, the memory is initialized by ngx_pool_t * ngx_create_pool (size_t size, ngx_log_t * log); // creates the memory pool void ngx_destroy_pool (ngx_pool_t * Pool); // destroys the memory pool, including void ngx_reset_pool (ngx_pool_t * Pool); // resets the memory pool and releases void * ngx_palloc (ngx_pool_t * Pool, size_t size); // applies for memory, the memory goes through the alignvoid * ngx_pnalloc (ngx_pool_t * Pool, size_t size); // The applied memory. The memory is not aligned with void * ngx_pcalloc (ngx_pool_t * Pool, size_t size ); // apply for memory, the memory is aligned, and the applied memory is initialized void * ngx_pmemalign (ngx_pool_t * Pool, size_t size, size_t alignment); // apply for a large memory, the memory is aligned with ngx_int_t ngx_pfree (ngx_pool_t * Pool, void * P); // release all the large memory ngx_pool_cleanup_t * ngx_pool_cleanup_add (ngx_pool_t * P, size_t size ); // Add the cleanup size as size (data) void fill (ngx_pool_t * P, ngx_fd_t FD); // clear all cleanupvoid ngx_pool_cleanup_file (void * data); // close the file, data points to ngx_pool_cleanup_file_tvoid ngx_pool_delete_file (void * data); // deletes the file, and data points to ngx_pool_cleanup_file_t

2.1ngx _ alloc and ngx_calloc

The ngx_alloc and ngx_calloc functions are defined in src/OS/Unix/ngx_alloc. (h, C)

The ngx_alloc function is actually an encapsulation of the malloc function:

void *ngx_alloc(size_t size, ngx_log_t *log){    void  *p;    p = malloc(size);    if (p == NULL) {        ngx_log_error(NGX_LOG_EMERG, log, ngx_errno,                      "malloc(%uz) failed", size);    }    ngx_log_debug2(NGX_LOG_DEBUG_ALLOC, log, 0, "malloc: %p:%uz", p, size);    return p;}

The ngx_calloc function is the same as the ngx_alloc function, except for initializing the applied memory:

Void * ngx_calloc (size_t size, ngx_log_t * log) {void * P; P = ngx_alloc (size, log); If (p) {ngx_memzero (p, size ); // initialize the applied memory} return P ;}

2.2 create a memory pool

First look at a macro definition

#define NGX_POOL_ALIGNMENT       16

This macro defines the memory alignment size.

Memory Pool creation function:

Ngx_pool_t * ngx_create_pool (size_t size, ngx_log_t * log) {ngx_pool_t * P; P = ngx_memalign (ngx_pool_alignment, size, log); // The applied memory is aligned by 16, the memory size must be an integer multiple of 16 and greater than or equal to size if (P = NULL) {return NULL;} p-> D. last = (u_char *) P + sizeof (ngx_pool_t); // point to the starting position of available memory, from which the memory is allocated to the user using p-> D. end = (u_char *) P + size; // point to the end position of available memory p-> D. next = NULL; P-> D. failed = 0; size = size-sizeof (ngx_pool_t); P-> max = (size <ng X_max_alloc_from_pool )? Size: ngx_max_alloc_from_pool; P-> current = P; // point to its own memory pool p-> chain = NULL; P-> large = NULL; P-> cleanup = NULL; p-> log = log; return P ;}

Look at the ngx_memalign function, which is in src/OS/Unix/ngx_alloc. (h, C ):

#if (NGX_HAVE_POSIX_MEMALIGN)void *ngx_memalign(size_t alignment, size_t size, ngx_log_t *log){    void  *p;    int    err;    err = posix_memalign(&p, alignment, size);    if (err) {        ngx_log_error(NGX_LOG_EMERG, log, err,                      "posix_memalign(%uz, %uz) failed", alignment, size);        p = NULL;    }    ngx_log_debug3(NGX_LOG_DEBUG_ALLOC, log, 0,                   "posix_memalign: %p:%uz @%uz", p, size, alignment);    return p;}#elif (NGX_HAVE_MEMALIGN)void *ngx_memalign(size_t alignment, size_t size, ngx_log_t *log){    void  *p;    p = memalign(alignment, size);    if (p == NULL) {        ngx_log_error(NGX_LOG_EMERG, log, ngx_errno,                      "memalign(%uz, %uz) failed", alignment, size);    }    ngx_log_debug3(NGX_LOG_DEBUG_ALLOC, log, 0,                   "memalign: %p:%uz @%uz", p, size, alignment);    return p;}#endif

Here is a brief introduction to the memalign function.

In the GNU system, the memory block address returned by malloc or realloc is a multiple of 8 (if it is a 64-bit system, it is a multiple of 16 ). If you need more granularity, use memalign or valloc. These functions are declared in the header file "stdlib. H.

In the GNU Library, you can use the free function to release memory blocks returned by memalign and valloc. But it cannot be used in the BSD system, and the BSD system does not provide a way to release such memory blocks.

Function: void * memalign (size_t boundary, size_t size)
The memalign function allocates a memory block with the size specified and the address multiple of boundary. The boundary parameter must be a power of 2! The memalign function can allocate large memory blocks and specify the granularity for the returned addresses.

Function: void * valloc (size_t size)
Using the valloc function is similar to using the memalign function. In the internal implementation of the valloc function, the page size is used as the alignment length and memalign is used to allocate memory. Its implementation is as follows:

void *valloc (size_t size){    return memalign (getpagesize (), size);} 

2.3 memory pool destruction Function

The ngx_destroy_pool function consists of three steps:
First, clear the memory of the ngx_pool_cleanup_t structure. This structure is mounted with a lot of hander cleanup
Then clear the large block memory, that is, the large block memory stored in the ngx_poll_large_t struct.
Finally, the ngx_pool_t memory is cleared, that is, the memory applied by ngx_create_pool.

Voidngx_destroy_pool (ngx_pool_t * Pool) {ngx_pool_t * P, * n; then * l; then * C; // call the cleaning handle mounted on ngx_pool_clean_t in the first step to clear the corresponding data, or close the file, socket connection, and so on for (C = pool-> cleanup; C = C-> next) {If (c-> handler) {ngx_log_debug1 (ngx_log_debug_alloc, pool-> log, 0, "Run cleanup: % P", c); C-> handler (c-> data) ;}// Step 2 releases large memory blocks, actually the memory applied by the ngx_palloc_large function for (L = pool-> large; L = L-> next) {ngx_log_debug1 (ngx_log_debug_alloc, pool-> log, 0, "Free: % P", L-> alloc); If (L-> alloc) {ngx_free (L-> alloc) ;}# if (ngx_debug) /** we cocould allocate the pool-> log from this pool * so we cannot use this log while free () ing the pool */For (P = pool, N = pool-> D. next;/* void */; P = n, n = N-> D. next) {ngx_log_debug2 (ngx_log_debug_alloc, pool-> log, 0, "Free: % P, unused: % uz", p, p-> D. end-p-> D. last); If (n = NULL) {break ;}# endif // step 3 release the memory applied by ngx_create_pool for (P = pool, n = pool-> D. next;/* void */; P = n, n = N-> D. next) {ngx_free (p); If (n = NULL) {break ;}}}

2.4 reset Memory Pool

The reset memory pool operation restores the memory pool to its initial state, that is, the memory is not allocated to the user. This operation releases large memory blocks,
To restore a small block of memory to its original state, you can modify the value of the last member in ngx_pool_data_t to point it to the initial allocable
Memory location for the user.

Voidngx_reset_pool (ngx_pool_t * Pool) {ngx_pool_t * P; ngx_pool_large_t * l; // release large memory for (L = pool-> large; L = L-> next) {If (L-> alloc) {ngx_free (L-> alloc) ;}} pool-> large = NULL; For (P = pool; P; P = p-> D. next) {P-> D. last = (u_char *) P + sizeof (ngx_pool_t); // Modify the last pointer so that it points to the initial allocable memory position }}

2.5 Memory Allocation Function

The memory allocation functions are ngx_palloc, ngx_pnalloc, and ngx_palloc_block and ngx_palloc_large for external user entries.

2.5.1 ngx_palloc and ngx_pnalloc

The difference between the two functions is that ngx_palloc allocates memory aligned with ngx_alignment from the pool memory pool, while ngx_pnalloc allocates memory suitable for size, regardless of memory alignment. Here we only look at the ngx_palloc function:

Void * ngx_palloc (ngx_pool_t * Pool, size_t size) {u_char * m; ngx_pool_t * P; // The applied memory is smaller than Max if (size <= pool-> MAX) {P = pool-> current; // In the memory pool linked list, check for idle memory do with size> = {// execute the alignment operation, // start with last, calculate the offset position pointer M = ngx_align_ptr (p-> D. last, ngx_alignment); If (size_t) (p-> D. end-m)> = size) {// find the satisfied memory block p-> D. last = m + size; return m;} p = p-> D. next;} while (p); // if no memory> = size is found in the memory pool linked list, call the ngx_palloc_block function to apply for the memory and connect the memory to the final return ngx_palloc_block (pool, size) of the memory pool linked list.} // The applied memory is greater than Max return ngx_palloc_large (pool, size );}

If the size <Max and no memory with the idle memory greater than or equal to the size is found in the memory pool linked list, the ngx_palloc_block is called.

Then let's take a look at ngx_palloc_block:

Static void * ngx_palloc_block (ngx_pool_t * Pool, size_t size) {u_char * m; size_t psize; ngx_pool_t * P, * New, * Current; psize = (size_t) (pool-> D. end-(u_char *) pool); // execute memory allocation by ngx_pool_alignment aligment. M = ngx_memalign (ngx_pool_alignment, psize, pool-> log); If (M = NULL) {return NULL;} new = (ngx_pool_t *) m; // initialize block new-> D. end = m + psize; New-> D. next = NULL; New-> D. failed = 0; m + = sizeof (NG X_pool_data_t); // M points to the starting position of the allocable memory. M = ngx_align_ptr (M, ngx_alignment); // memory alignment New-> D. last = m + size; // last point to the next allocation location, because the memory from m to m + size has been allocated current = pool-> current; For (P = current; p-> D. next; P = p-> D. next) {If (p-> D. failed ++> 4) {current = p-> D. next; // move current. Four failures indicate that the memory block space available in the previous step is small, and the memory will be allocated directly from the current position.} p-> D. next = new; // connect the newly allocated block to the memory pool-> current = current? Current: New; // return m if current is empty ;}

Note: ngx_palloc_block is a static function, indicating that this function can only be used in the current file and cannot be called by external users.

If the applied size is greater than Max, the user needs a large memory and calls ngx_palloc_large to allocate the memory to the user.

Ngx_palloc_large:

Static void * ngx_palloc_large (ngx_pool_t * Pool, size_t size) {void * P; ngx_uint_t N; ngx_pool_large_t * large; P = ngx_alloc (size, pool-> log ); // apply for memory if (P = NULL) {return NULL;} n = 0; // The following lines link the allocated memory to the large chain of the pool, // This indicates that the original pool has previously allocated large memory. For (large = pool-> large; large = Large-> next) {If (large-> alloc = NULL) {large-> alloc = P; return P;} If (N ++> 3) {break ;}// if large memory is not allocated before the pool, then there is no ngx_pool_large_t to manage the large memory. // execute the allocation of the ngx_pool_large_t struct to manage the large memory blocks. Large = ngx_palloc (pool, sizeof (ngx_pool_large_t); If (large = NULL) {ngx_free (p); return NULL;} large-> alloc = P; large-> next = pool-> large; pool-> large = large; return P ;}

Note: This is a static function, indicating that external functions are not called at will, but are provided for internal allocation calls, that is, when nginx needs to allocate memory, it does not determine whether it is a large memory or a small memory. Instead, it is determined by the memory allocation function, which is completely transparent to user requirements.

2.5.2 ngx_pcalloc, ngx_free, ngx_pmemalign

Ngx_pcalloc is almost the same as ngx_palloc, but the allocated memory is initialized.

Ngx_pcalloc:

Void * ngx_pcalloc (ngx_pool_t * Pool, size_t size) {void * P; P = ngx_palloc (pool, size); If (p) {ngx_memzero (p, size ); // initialization} return P ;}

Note the following before using the ngx_free function:

ngx_int_tngx_pfree(ngx_pool_t *pool, void *p){    ngx_pool_large_t  *l;    for (l = pool->large; l; l = l->next) {        if (p == l->alloc) {            ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0,                           "free: %p", l->alloc);            ngx_free(l->alloc);            l->alloc = NULL;            return NGX_OK;        }    }    return NGX_DECLINED;}

Ngx_pmemalign allocates the size of memory and performs alignment-based alignment. Then, it is mounted to the large field and processed as a large memory.

void *ngx_pmemalign(ngx_pool_t *pool, size_t size, size_t alignment){    void              *p;    ngx_pool_large_t  *large;    p = ngx_memalign(alignment, size, pool->log);    if (p == NULL) {        return NULL;    }    large = ngx_palloc(pool, sizeof(ngx_pool_large_t));    if (large == NULL) {        ngx_free(p);        return NULL;    }    large->alloc = p;    large->next = pool->large;    pool->large = large;    return p;}

Note that this function is applied for large memory.

2.6 ngx_clean_add

This function is used to add a memory block and hander function to cleanup. However, this handler must be set by ourselves because the initial value is empty.

ngx_pool_cleanup_t *ngx_pool_cleanup_add(ngx_pool_t *p, size_t size){    ngx_pool_cleanup_t  *c;    c = ngx_palloc(p, sizeof(ngx_pool_cleanup_t));    if (c == NULL) {        return NULL;    }    if (size) {        c->data = ngx_palloc(p, size);        if (c->data == NULL) {            return NULL;        }    } else {        c->data = NULL;    }    c->handler = NULL;    c->next = p->cleanup;    p->cleanup = c;    ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, p->log, 0, "add cleanup: %p", c);    return c;}

Http://blog.csdn.net/xiaoliangsky/article/details/39523875

Work hard

Nginx learns four memory pool ngx_pool_t and memory management operations