MySQL series: Memory Management for innodb Engine Analysis

MySQL series: Memory Management for innodb Engine Analysis

Innodb implements its own memory pool system and memory heap allocation system. In the innodb memory management system, there are roughly three parts: basic memory block allocation management, memory partner allocation, and memory heap allocation. The main purpose of innodb to define and implement a memory pool is to provide memory usage and efficiency, preventing memory fragmentation and memory allocation tracking and debugging. Let's take a look at their relationships and structures.

The link structure is as follows:

In:

Ut_mem_block is the basic memory management.

Buddy allocatZ success? Http://www.bkjia.com/kf/ware/vc/ "target =" _ blank "class =" keylink "> keys + MS67 + keys + 38 zltcTKtc/keys + 7W2NKqtcS + keys/Co7oKPHByZSBjbGFzcz0 =" brush: SQL; "> typedef ut_mem_block_struct {ulint size;/* memory size of the allocated block */ulint magic_n;/* node magic word, used for verification */UT_LIST_NODE_T (ut_mem_block_t) mem_block_list; /* block list node, specify prev node and next node */};The list definition of a block is a global variable. UT_LIST_BASE_NODE_T (ut_mem_block_t) ut_mem_block_list; all allocated blocks are added to this list. When the ut_malloc_low function allocates memory, it adds the allocated block to the list. When ut_free is used, the block where the memory released by the club is deleted from the list. In addition to these two functions, innodb also provides the ut_free_all_mem function to release all allocated blocks and calculate the total number of allocated memory ut_total_allocated_memory.
The basic memory management method is as follows:
Ut_malloc_low allocates an n-length memory block and records the allocated block to ut_mem_block_list.
Ut_malloc has the same function as ut_malloc_low, but the allocated memory is initialized with 0.
Ut_free releases an allocated memory block and deletes it from ut_mem_block_list.
Ut_free_all_mem: release all memory blocks in ut_mem_block_list and clear ut_mem_block_list.
The above functions support multi-threaded concurrent operations, that is, thread security. The purpose of innodb is to ensure that all malloc memory is in ut_mem_block_list for management. The basic memory management structure is as follows:
2. the partner distributor of innodb is a memory manager based on the ut_malloc_low function. When creating a partner distributor, innodb will use ut_malloc_low to open up a large memory block, then, allocate the memory for this block with the partner. The partner distributor of innodb is a management method based on the base of 2. Its buddy alloc pool is defined as follows:

Struct mem_pool_struct {byte * buf;/* overall memory handle */ulint size;/* overall memory size */ulint reserved; /* total memory size allocated currently */mutex;/* multi-thread mutex */UT_LIST_BASE_NODE_T (mem_area_t) free_list [64];/* area_t linked list array, each array unit can manage the list of I-power memory blocks. I is the subscript of the array */};

Struct mem_area_struct {ulintsize_and_free;/* the memory size of the area (must be the power of 2). The last bit indicates whether it has been released */UT_LIST_NODE_T (mem_area_t) free_list; /* the upper and lower areas of the area linked list, because buddy area is split, there may be multiple */};

Mem_area_t is the memory area of a buddy, that is, mem_area_struct. The following is a buddy list distribution for 32-bit machines to manage 1024-byte memory blocks:
Each area is determined by the mem_area_t header and the allocable memory (memory_buffer). The length of memory_buffer is not less than the length of mem_area_t header, on a 32-bit machine, the mem_area_t header should be 16 bytes (8-byte alignment ).

2.1mem _ area_t splitting may cause mem_area_t splitting during memory allocation. In the above example, we need to allocate a 200-byte memory, at this time, the distribution process of the partner distributor is as follows: 1. locate the number (200) of the I power nearest 2 to 256 + sizeof (mem_area_t), and confirm that I = 8, 2. in the free_list [I] list, check whether any idle node exists. If yes, set the node position to no free. if no, perform a search for available memory on the I + 1 layer. in the above example, I + 1 = 9, free_list is empty, continue searching at the I + 2 layer, and so on until the node layer is found, that is, I = 10; 4. split Layer 10 into two 512-sized 9th-level 9th-level nodes, delete the area from layer 10, and add two 512-level nodes to the layer. 5. split the first node in layer 9th, split two layer 256 nodes with a size of 8th, delete them from the ninth layer, and add Two nodes. 6. Allocate the first 256-size area to the corresponding operator and set it to no free logo. A 200-byte memory pool structure is allocated as follows:
If the allocated area_t is deleted from the free_list [I] linked list, it is recorded on buddy.
2.2mem _ area_t merging if 200 bytes are allocated, the data will be returned to buddy allocator after use. Take the preceding example as follows:
1. the user returns the memory allocated by the partner. The user first finds his buddy Based on the area_t information, that is, another unallocated area in layer 8th.
2. Find the buddy area and check whether the buddy area is in the release state. If yes, trigger the merge and delete the buddy area from the 8th layer to form a 512-size 9th layer area,
3. Repeat 1 ~ In step 2, the user and the other buddy area on the ninth layer will be merged into a 1024-size 10th-layer area.

2.3 buddy allocator interface function: mem_pool_create constructs a buddy allocator
Mem_area_alloc uses buddy allocator to allocate a piece of memory.
Mem_area_free returns a piece of memory to buddy allocator.
Mem_pool_get_reserved: Obtain the memory size used by buddy allocator.

3 memory Allocation heap the memory management in innodb is embodied in the memory allocation and management of mem_heap_t. All operations on memory allocation call the mem_heap API method, the structure of mem_heap_t is defined as follows:

Struct mem_block_info_struct {ulint magic_n;/* Magic word */char file_name [8];/* Memory allocated file */ulint line; /* row of the file with memory allocation */ulint len;/* block length */ulint type;/* underlying allocation type dependent on, three types are available: DYNAMIC, BUFFER, and BTR_SEARCH */ibool init_block;/* Whether it is an externally allocated memory block */ulint free;/* the space occupied */ulint start; /* start position of the allocable memory */byte * free_block;/* Backup block, which is available only in the BTR_SEARCH mode */UT_LIST_BASE_NODE_T (mem_block_t) base; UT_LIST_NODE_T (mem_block_t) list ;};

Note: mem_block_info_struct/mem_block_info_t/mem_block_t/mem_heap_t is equivalent.
The memory structure of mem_heap_t is as follows:
The system's malloc and free are used for memory management. MySQL uses the system memory management mode by default. Some experienced DBAs use the system memory management mode + TMalloc for memory optimization, and use TMalloc for efficient memory management to improve MySQL performance.