The memory pool system and the memory heap allocation system are implemented in InnoDB, which is divided into three parts in InnoDB memory management system: basic memory block allocation management, memory partner allocator and memory heap allocator. InnoDB The primary purpose of defining and implementing a memory pool is to provide memory usage and efficiency to prevent memory fragmentation and memory allocation tracking and debugging. Let's take a look at their relationship and structure first.
The following is a diagram of its relationship structure:
In the following:
Ut_mem_block blocks are basic memory management
Buddy allocator is a memory partner allocator
Mem_heap is the memory heap allocator
1. Basic Memory Managementmemory allocation and memory release in InnoDB are managed through a unified architecture, specifically in ut0mem.h and UT0MEM.C, the most important of which is the encapsulation of malloc and free. By using a list structure to manage the allocated memory, the structure is as follows:
typedef ut_mem_block_struct { ulint size; /* The memory size of the allocated block */ ulint magic_n; /* Node magic word, used to verify the use of * /ut_list_node_t (ut_mem_block_t) mem_block_list; /*block list node, specifying prev node and next node*/ };
the list definition of block is a global variable, ut_list_base_node_t (ut_mem_block_t) ut_mem_block_list; All allocated blocks will be added to the list. When the Ut_malloc_low function allocates memory, the allocated block is added to the list, and the block from which the club frees memory is removed from the list when ut_free. In addition to these two functions, InnoDB also provides the UT_FREE_ALL_MEM function to release the total number of allocated blocks and statistics allocated memory ut_total_allocated_memory function.
basic memory management methods are as follows:
Ut_malloc_low to allocate an n-length block of memory and record the allocated blocks into ut_mem_block_list.
Ut_malloc &NB Sp is the same as Ut_malloc_low, but the allocated memory is initialized with 0.
Ut_free &NBSP ; releases an allocated block of memory and removes it from the ut_mem_block_list.
UT_FREE_ALL_MEM release Ut_mem_block_lis T all memory blocks and empty ut_mem_block_list
The above functions support multithreading concurrency, which means thread-safe. The purpose of this innodb is to ensure that all malloc's memory isUt_mem_block_list in order to manage. the underlying memory management structure is as follows:
2. Partner DistributorInnoDB's partner allocator is a memory manager based on the Ut_malloc_low function, and when the partner allocator is created, InnoDB uses Ut_malloc_low to open up a large block of memory and allocates the memory usage of the block with the partner assignment. InnoDB's partner allocator is based on a 2 base management approach, and its buddy Alloc pool is defined as follows:
struct mem_pool_struct { byte* buf; /* Handle of overall memory */ ulint size; /* Overall memory size */ ulint reserved; /* Total memory size currently allocated */ mutex mutex; /* Multithreading Mutex * /ut_list_base_node_t (mem_area_t) free_list[64]; /*area_t array, each array cell can manage 2 of the I-memory block list, I is the array subscript */ };
struct mem_area_struct { ulintsize_and_free; /*area memory Size (must be 2 of the second side), the last bit indicates whether /ut_list_node_t (mem_area_t) Free_list has been released; /*area List of the upper and lower area, because the buddy area will be split, there may be multiple */ };
mem_area_t is a buddy area of memory, Mem_area_struct. The following is a 32-bit machine that manages 1024-byte memory blocks of the buddy list distribution:
Each area is determined by the mem_area_t head and the assignable memory (Memory_buffer), the length of the memory_buffer is not less than the length of the mem_area_t head, The mem_area_t header on a 32-bit machine should be 16 bytes (8 byte aligned).
The Division of 2.1mem_area_tIn the memory allocation process, it is possible to cause mem_area_t division, or in the above example, join us to allocate a 200 bytes of memory, this time the allocation process of the partner allocator is this: 1. Find a 200+sizeof (mem_area_t) The nearest 2 i-th-square number (256), OK i = 8, 2. InFree_list[i] In the list to find out if there is an idle node, and if so, will node position no free. If not, perform a lookup on the i + 1 layer to see if there is memory available,3. In the example above, I+1=9,free_list is empty and continues to be found at the i+2 layer, one analogy at a time until a layer with node is found, that is, i = ten;4. First split the 10 layer, splitting it into a 2,512-size 9th node and removing area from 10 and adding 2 512 nodes to the 9th layer.5. Then split the first node of the 9th layer, splitting two 8th-level nodes with a size of 256, and deleting from the nineth layer, adding 2 nodes to the 8th layer. 6. Assign the area of the first 256 size to the corresponding operator and set it to the no free sign. The following is a 200-byte memory pool structure assigned:
if the assigned area_t is removed from the free_list[i] linked list, that means it will be recorded on Buddy.
2.2mem_area_t Merge if 200 bytes are allocated, the use will be returned to Buddy allocator, or the above example, the area merge will occur, the steps are as follows:
1. The user returns the memory allocated by the partner, first based on area_t information to find their own buddy, that is, the 8th layer is not assigned to the area.
2. After locating the buddy area, determine if the Buddy area is released, and if so, trigger the merge to remove itself and buddy area from layer 8th, merging into a 512-size 9th area,
3. Repeat 1 to 2 steps and merge yourself with another buddy area on the nineth layer into a 1024-size 10th area.
2.3buddy Allocator interface function: Mem_pool_create build a buddy allocator
Mem_area_alloc allocating a piece of memory with Buddy allocator
Mem_area_free returns a piece of memory to Buddy allocator
Mem_pool_get_reserved get buddy allocator already used memory size
3 Memory allocation heap (storage heap)The final embodiment of memory management in InnoDB is mem_heap_t memory allocation and management, and all operations on memory allocation call Mem_heap's API method, and the structure of mem_heap_t is defined as follows:
struct mem_block_info_struct{ ulint magic_n; /* Magic word */ char file_name[8]; /* Files allocated in memory */ ulint line; /* The file where the memory is allocated */ ulint len; Length of/*block */ ulint type; /* dependent on the underlying allocation type, there is the dynamic, BUFFER, btr_search three types */ Ibool init_block;/* is the external allocated memory block */ ulint free; /* Occupied space size */ ulint start; /* Starting position for allocated memory */ byte* free_block; /* Standby block, only available in Btr_search mode * /ut_list_base_node_t (mem_block_t) BASE; ut_list_node_t (mem_block_t) LIST;};
Note :mem_block_info_struct/mem_block_info_t/mem_block_t/mem_heap_t is equivalent
The memory structure of mem_heap_t is as follows:
A few points about mem_heap_t:
1. A mem_block_t minimum space of not less than 64 bytes, the standard size is 8KB, the space allocated in non-mem_heap_buffer mode is not more than page size-200 (page size is generally 16KB)
There are three types of 2.mem_heap_t, namely dynamic, BUFFER, and Btr_search, which are mem_block_t distributed in dynamic mode based on Buddy allocator, and in Btr_search mode, Using Free_block as memory allocation is more complex in buffer mode, if you allocate memory size < page size half, use Buddy Alloc, otherwise use Buf_frame memory allocation method (this belongs to Buf0buf. XX inside the way, has not begun to analyze).
3.mem_heap_t when assigning a new mem_block_t, it must be twice times the size of the last node of the heap allocated, if the size of the allocation exceeds mem_max_alloc_in_buf (equivalent to a page size), the heap type is determined, In not dynamic mode, the maximum is a mem_max_alloc_in_buf size. If the other mode is set to mem_block_standard_size standard size, outside of these limits, if you need to allocate more memory than these limits, allocate the memory size to the mem_block_t allocation. The assigned mem_block_t is always added to the end of the heap base list, which is the top of the heap stack.
4.mem_heap_t is always released from the top when releasing mem_block_t, until it cannot be released (mem_block_t is not returned by the user). When Mem_block_t is released, it is also necessary to refer to the dynamic, BUFFER, and btr_search types for relative restitution rules (and 2 points are the opposite).
mem_heap_t Function Method Description:
Mem_heap_create creating a mem_heap_t with dynamic mode
Mem_heap_create_in_buffer creating a mem_heap_t with buffer mode
Mem_heap_create_in_btr_search creating a mem_heap_t with Btr_search mode
Mem_heap_free Releasing mem_heap_t objects
Mem_alloc is created in mem_heap_dynamic mode and allocates a piece of memory of a specified size (in this way mem_heap_t will only have one mem_ block_t)
Mem_free returns mem_heap_t allocated memory and releases mem_heap_t
Mem_heap_alloc allocating a piece of memory on the specified mem_heap_t
Mem_heap_get_heap_top get the address of the heap top block to use memory
Mem_heap_empty emptying the specified mem_heap_t
Mem_heap_get_top gets the mem_block_t pointer of the specified n size at the top of the heap
Mem_heap_free_top release heap N-size mem_block_t block at top
4 Summary InnoDB provides a memory pool and heap allocation method to manage memory uniformly, the main purpose is to increase the memory rate. In the MySQL-5.6 version, InnoDB offers two choices, one is to use the memory pool provided by InnoDB to manage memory, and one is to provide system malloc and free as memory management. MySQL defaults to the way the system manages memory, and some experienced DBAs use the system's managed memory mode +tmalloc to do memory optimization and improve MySQL performance with tmalloc efficient memory management.
MySQL series: memory management of INNODB engine analysis
