MySQL series: Memory Management for innodb Engine analysis, mysqlinnodb
Innodb implements its own memory pool system and memory heap allocation system. In the innodb memory management system, there are roughly three parts: basic memory block allocation management, memory partner allocation, and memory heap allocation. The main purpose of innodb to define and implement a memory pool is to provide memory usage and efficiency, preventing memory fragmentation and memory allocation tracking and debugging. Let's take a look at their relationships and structures.
The link structure is as follows:
In:
Ut_mem_block is the basic memory management.
Buddy allocator is a memory partner distributor.
Mem_heap is a memory heap distributor.
1. basic memory management: Memory Allocation and memory release in innodb are managed in a unified structure. The specific implementation is in ut0mem. h and ut0mem. among c, the most important is the encapsulation of malloc and free. A linked list struct is used to manage allocated memory. The struct is as follows:
Typedef ut_mem_block_struct {ulint size;/* memory size of the allocated block */ulint magic_n;/* node magic word used for verification */UT_LIST_NODE_T (ut_mem_block_t) mem_block_list; /* block list node, specify prev node and next node */};The list definition of a block is a global variable. UT_LIST_BASE_NODE_T (ut_mem_block_t) ut_mem_block_list; all allocated blocks are added to this list. When the ut_malloc_low function allocates memory, it adds the allocated block to the list. When ut_free is used, the block where the memory released by the club is deleted from the list. In addition to these two functions, innodb also provides the ut_free_all_mem function to release all allocated blocks and calculate the total number of allocated memory ut_total_allocated_memory.
The basic memory management method is as follows:
Ut_malloc_low allocates an n-length memory block and records the allocated block to ut_mem_block_list.
Ut_malloc has the same function as ut_malloc_low, but the allocated memory is initialized with 0.
Ut_free releases an allocated memory block and deletes it from ut_mem_block_list.
Ut_free_all_mem: release all memory blocks in ut_mem_block_list and clear ut_mem_block_list.
The above functions support multi-threaded concurrent operations, that is, thread security. The purpose of innodb is to ensure that all malloc memory is in ut_mem_block_list for management. The basic memory management structure is as follows:
2. the partner distributor of innodb is a memory manager based on the ut_malloc_low function. When creating a partner distributor, innodb will use ut_malloc_low to open up a large memory block, then, allocate the memory for this block with the partner. The partner distributor of innodb is a management method based on the base of 2. Its buddy alloc pool is defined as follows:
Struct mem_pool_struct {byte * buf;/* overall memory handle */ulint size;/* overall memory size */ulint reserved; /* total memory size allocated currently */mutex;/* multi-thread mutex */UT_LIST_BASE_NODE_T (mem_area_t) free_list [64];/* area_t linked list array, each array unit can manage the list of I-power memory blocks. I is the subscript of the array */};
Struct mem_area_struct {ulintsize_and_free;/* the memory size of the area (must be the power of 2). The last bit indicates whether it has been released */UT_LIST_NODE_T (mem_area_t) free_list; /* the upper and lower areas of the area linked list, because buddy area is split, there may be multiple */};Mem_area_t is the memory area of a buddy, that is, mem_area_struct. The following is a buddy list distribution for 32-bit machines to manage 1024-byte memory blocks:
Each area is determined by the mem_area_t header and the allocable memory (memory_buffer). The length of memory_buffer is not less than the length of mem_area_t header, on a 32-bit machine, the mem_area_t header should be 16 bytes (8-byte alignment ).
2.1mem _ area_t splitting may cause mem_area_t splitting during memory allocation. In the above example, we need to allocate a 200-byte memory, at this time, the distribution process of the partner distributor is as follows: 1. locate the number (200) of the I power nearest 2 to 256 + sizeof (mem_area_t), and confirm that I = 8, 2. in the free_list [I] list, check whether any idle node exists. If yes, set the node position to no free. if no, perform a search for available memory on the I + 1 layer. in the above example, I + 1 = 9, free_list is empty, continue searching at the I + 2 layer, and so on until the node layer is found, that is, I = 10; 4. split Layer 10 into two 512-sized 9th-level 9th-level nodes, delete the area from layer 10, and add two 512-level nodes to the layer. 5. split the first node in layer 9th, split two layer 256 nodes with a size of 8th, delete them from the ninth layer, and add Two nodes. 6. Allocate the first 256-size area to the corresponding operator and set it to no free logo. A 200-byte memory pool structure is allocated as follows:
If the allocated area_t is deleted from the free_list [I] linked list, it is recorded on buddy.
2.2mem _ area_t merging if 200 bytes are allocated, the data will be returned to buddy allocator after use. Take the preceding example as follows:
1. the user returns the memory allocated by the partner. The user first finds his buddy Based on the area_t information, that is, another unallocated area in layer 8th.
2. Find the buddy area and check whether the buddy area is in the release state. If yes, trigger the merge and delete the buddy area from the 8th layer to form a 512-size 9th layer area,
3. Repeat 1 ~ In step 2, the user and the other buddy area on the ninth layer will be merged into a 1024-size 10th-layer area.
2.3 buddy allocator interface function: mem_pool_create constructs a buddy allocator
Mem_area_alloc uses buddy allocator to allocate a piece of memory.
Mem_area_free returns a piece of memory to buddy allocator.
Mem_pool_get_reserved: Obtain the memory size used by buddy allocator.
3 memory Allocation heap the memory management in innodb is embodied in the memory allocation and management of mem_heap_t. All operations on memory allocation call the mem_heap API method, the structure of mem_heap_t is defined as follows:
Struct mem_block_info_struct {ulint magic_n;/* Magic word */char file_name [8];/* Memory allocated file */ulint line; /* row of the file with memory allocation */ulint len;/* block length */ulint type;/* underlying allocation type dependent on, three types are available: DYNAMIC, BUFFER, and BTR_SEARCH */ibool init_block;/* Whether it is an externally allocated memory block */ulint free;/* the space occupied */ulint start; /* start position of the allocable memory */byte * free_block;/* Backup block, which is available only in the BTR_SEARCH mode */UT_LIST_BASE_NODE_T (mem_block_t) base; UT_LIST_NODE_T (mem_block_t) list ;};
Note: mem_block_info_struct/mem_block_info_t/mem_block_t/mem_heap_t is equivalent.
The memory structure of mem_heap_t is as follows:
Several key points about mem_heap_t:
1. The minimum size of a mem_block_t is not less than 64 bytes, and the standard size is 8 KB. In non-MEM_HEAP_BUFFER mode, the allocated space is not greater than page size-200 (page size is generally 16 KB)
2. mem_heap_t has three types: DYNAMIC, BUFFER, and BTR_SEARCH. In DYNAMIC mode, mem_block_t is allocated based on buddy allocator. In BTR_SEARCH mode, free_block is used for memory allocation, in BUFFER mode, it is complicated. If the size of the allocated memory is less than half of the page size, buddy alloc is used; otherwise, buf_frame memory allocation is used (this is a buf0buf. the method in XX has not been analyzed yet ).
3. when mem_heap_t allocates a new mem_block_t, it must be twice the size of the last heap node. If the allocated size exceeds MEM_MAX_ALLOC_IN_BUF (equivalent to a page size), the heap type is determined, in non-DYNAMIC mode, the maximum value is the size of a MEM_MAX_ALLOC_IN_BUF. If MEM_BLOCK_STANDARD_SIZE is set in other modes, in addition to these restrictions, if the memory to be allocated exceeds these limits, mem_block_t is allocated based on the size of the allocated memory. The allocated mem_block_t is always added to the end of the heap base list, that is, the top of the heap stack.
4. When mem_heap_t is released, it is always released from the top until it cannot be released (mem_block_t is not returned by the user ). When mem_block_t is released, it is also necessary to refer to the return rules for DYNAMIC, BUFFER, and BTR_SEARCH types (which correspond to the two points ).
Mem_heap_t Function Method description:
Mem_heap_create creates a mem_heap_t in DYNAMIC mode.
Mem_heap_create_in_buffer creates a mem_heap_t in BUFFER mode.
Mem_heap_create_in_btr_search creates a mem_heap_t instance in BTR_SEARCH mode.
Mem_heap_free releases the mem_heap_t object.
Mem_alloc is created in MEM_HEAP_DYNAMIC mode and allocated with a memory of the specified size (in this mode, mem_heap_t only has one mem_block_t)
Mem_free returns the memory allocated by mem_heap_t and releases mem_heap_t.
Mem_heap_alloc allocates a block of memory on the specified mem_heap_t.
Mem_heap_get_heap_top obtains the memory address available for the heap top block.
Mem_heap_empty clears the specified mem_heap_t
Mem_heap_get_top obtains the specified n-size mem_block_t pointer at the top of the heap.
Mem_heap_free_top releases N-sized mem_block_t block on the top of heap.
4. Summary innodb provides a memory pool and heap allocation method to manage the memory in a unified manner. The main purpose is to increase the memory rate. In the MySQL-5.6 version, innodb provides two options: one is to use the memory pool provided by innodb to manage the memory, and the other is to provide the system malloc and free for memory management. MySQL uses the system memory management mode by default. Some experienced DBAs use the system memory management mode + TMalloc for memory optimization, and use TMalloc for efficient memory management to improve MySQL performance.
