MySQL series: InnoDB source code Analysis of memory management

The memory pool system and the memory heap allocation system are implemented in InnoDB, which is divided into three parts in InnoDB memory management system: basic memory block allocation management, memory partner allocator and memory heap allocator. InnoDB The primary purpose of defining and implementing a memory pool is to provide memory usage and efficiency to prevent memory fragmentation and memory allocation tracking and debugging. Let's take a look at their relationship and structure first.

Here is a diagram of its relationship structure:

In the following:

Ut_mem_block blocks are basic memory management

Buddy allocator is a memory partner allocator

Mem_heap is the memory heap allocator

1. Basic Memory ManagementMemory allocation and memory release in InnoDB are managed through a unified structure, with detailed realities now ut0mem.h and UT0MEM.C. The most important of these is the encapsulation of malloc and free.

The allocated memory is managed through a list structure. Structural bodies such as the following:

typedef ut_mem_block_struct {            ulint        size;                                   /* The memory size of the allocated block */            ulint        magic_n;                                /* Node magic word, used to verify the use of *            /ut_list_node_t (ut_mem_block_t) mem_block_list;       /*block list node, specifying prev node and next node*/  };

The list definition of a block is a global variable. ut_list_base_node_t (ut_mem_block_t) ut_mem_block_list; All allocated blocks will be added to this LIST.

When the Ut_malloc_low function allocates memory, the allocated block is added to the list. At the time of the Ut_free, the block of memory released by the clubhouse is removed from the list.

In addition to these two functions, InnoDB also provides the UT_FREE_ALL_MEM function to release the total number of allocated blocks and statistics allocated memory ut_total_allocated_memory function.

  basic memory management methods such as the following:
        Ut_malloc_low       & nbsp             allocates an n-length block of memory and records the allocated blocks into ut_mem_block_list.
        Ut_malloc                       &NB Sp     with the Ut_malloc_low function, the allocated memory is initialized with 0.
        Ut_free                       &NBSP ;         frees up an allocated block of memory and removes it from the ut_mem_block_list.

Ut_free_all_mem release ut_mem_block_list all memory blocks and empty ut_mem_block_list
The above functions are supported for multithreading concurrent operations. This means that it is thread-safe. The purpose of InnoDB is to ensure that all of malloc's memory is ut_mem_block_list in order to manage it . The structure of the underlying memory management is as follows:
2. Partner Allocator InnoDB's partner allocator is a memory manager based on the Ut_malloc_low function, and when creating a partner allocator, InnoDB will open up a very large block of memory with Ut_malloc_low. The memory usage of this block is then allocated with the partner assignment.

InnoDB's partner allocator is based on a 2 base management approach, with its buddy Alloc pool definition such as the following:

   struct mem_pool_struct    {           byte*               buf;                          /* Handle of overall memory */        ulint                size;                        /* Overall memory size */        ulint                reserved;                    /* Total memory size currently allocated */        mutex             mutex;                          /* Multi-threaded mutual rejection amount *        /ut_list_base_node_t (mem_area_t) free_list[64];    An array of/*area_t arrays, each of which can manage a list of 2 I-memory blocks, I is the subscript of the array */    };

   struct mem_area_struct    {         ulintsize_and_free;                                        /*area memory Size (must be 2 of the second side), the last bit indicates whether         /ut_list_node_t (mem_area_t) Free_list has been released;       /*area List of the upper and lower area, because the buddy area will be split, there may be multiple */     };

mem_area_t is a buddy area of memory. That's mem_area_struct.

The following is a 32-bit machine that manages 1024-byte memory blocks of the buddy list distribution:

watermark/2/text/ahr0cdovl2jsb2cuy3nkbi5uzxqvexvhbnj4zhu=/font/5a6l5l2t/fontsize/400/fill/i0jbqkfcma==/ Dissolve/70/gravity/center ">
Each area is determined by a mem_area_t header and assignable memory (Memory_buffer), and the length of the memory_buffer is not less than The length of the mem_area_t header should be 16 bytes (8 byte aligned) on the 32-bit machine mem_area_t.

The Division of 2.1mem_area_t in the memory allocation process, it is possible to cause mem_area_t division. Or in the example above.        Increase our allocation of a 200 bytes of memory, this time the allocation process of the partner allocator is this: 1. Find a number (256) of the mem_area_t nearest 2 to 200+sizeof, ok i = 8. 2. In the list of free_list[i], find out if there is a spare node, fake, will node position no free. Suppose not. Run on the i + Layer 1 to find out if there is memory available. 3. In the example above, I+1=9,free_list is empty and continues to be found at the i+2 layer, one analogy. Until you find the layer with node, which is i = ten; 4. Divide the 10 layers first. Divide the 9th layer into 2,512-size nodes and remove area from 10. Add 2 512 node to the 9th layer. 5. Then split on the first node of the 9th layer, splitting two 8th-level nodes with a size of 256 and removing from the nineth layer, adding 2 nodes to the 8th layer. 6. Assign the area of the first 256 size to the appropriate operator and set it to the no free sign.

here is a 200-byte memory pool structure assigned:
assuming that the assigned area_t will be removed from the free_list[i] linked list, that is to say, the Buddy is recorded.
The merger of the 2.2mem_area_t assumes that the 200 bytes are allocated, and the completion will be returned to Buddy allocator, or the example above. An area merge scenario occurs, such as the following:
1. The user returns the memory allocated by the partner, first to find his own buddy based on Area_t's information, that is, the other area on the 8th floor that is not assigned.
2. After locating Buddy area. Infer whether the Buddy area is a release state, assuming yes, triggering the merge. Remove yourself and Buddy area from layer 8th, merging into a 512-size 9th area,
3. After 1 to 2 steps, the other buddy area of the Nineth layer is merged into a 1024-size 10th area.

2.3buddy Allocator interface function: Mem_pool_create build a buddy allocator
Mem_area_alloc allocating a piece of memory with Buddy allocator
Mem_area_free returns a piece of memory to Buddy allocator
Mem_pool_get_reserved get buddy allocator already used memory size

3 Memory allocation heap (storage heap)The memory management in InnoDB is finally manifested in the form of mem_heap_t memory allocation and management, and all the operations on memory allocation call Mem_heap's API method, MEM_HEAP_T's structure definition is as follows:

struct mem_block_info_struct{     ulint magic_n;         /* Magic word */     char file_name[8];    /* Files allocated in memory */     ulint line;            /* The file where the memory is allocated */     ulint len;             Length of/*block */     ulint type;            /* dependent on the underlying allocation type, there is the dynamic, BUFFER, btr_search three types */     Ibool init_block;/* is the external allocated memory block */     ulint free;           /* Occupied space size */     ulint start;         /* Starting position for allocated memory */     byte* free_block;     /* Standby block, only available in Btr_search mode *     /ut_list_base_node_t (mem_block_t) BASE;     ut_list_node_t (mem_block_t) LIST;};

Note :mem_block_info_struct/mem_block_info_t/mem_block_t/mem_heap_t is equivalent
The memory structure of the mem_heap_t is as follows:


A few points about mem_heap_t:
1. A mem_block_t minimum space of not less than 64 bytes, the standard size is 8KB, the space allocated in non-mem_heap_buffer mode is not more than page size-200 (page size is generally 16KB)

There are three types of 2.mem_heap_t, each of which is dynamic, BUFFER, Btr_search. In dynamic mode, the mem_block_t assignment is based on Buddy allocator. In Btr_search mode, Free_block is used as memory allocation, which is more complex in buffer mode. Assuming that the allocated memory size < page size is half, use Buddy Alloc, otherwise use Buf_frame memory allocation method (this belongs to Buf0buf. XX inside the way. has not yet started analysis).

3.mem_heap_t allocating a new mem_block_t must be twice times the size of the last node of the heap, assuming that the size of the allocation exceeds mem_max_alloc_in_buf (equivalent to a page size), the heap type is inferred, In not dynamic mode. The largest is a mem_max_alloc_in_buf size. Suppose the other mode is set to Mem_block_standard_size standard size. Outside of these limits, assume that the memory allocated is greater than these limits, and that the allocation of memory size is subject to mem_block_t allocations. The allocated mem_block_t always increases to the end of the heap base list, which is the top of the heap stack.

4.mem_heap_t is always released from the top when releasing mem_block_t, until it cannot be released (mem_block_t is not returned by the user). When Mem_block_t is released, it is also necessary to refer to the dynamic, BUFFER, and btr_search types for relative restitution rules (and 2 points are corresponding).

mem_heap_t Function Method Description:
Mem_heap_create creating a mem_heap_t with dynamic mode
Mem_heap_create_in_buffer creating a mem_heap_t with buffer mode
Mem_heap_create_in_btr_search creating a mem_heap_t with Btr_search mode
Mem_heap_free Releasing mem_heap_t objects
Mem_alloc is created in mem_heap_dynamic mode. and allocate a piece of memory of a specified size (in such a way mem_heap_t will only have a mem_block_t)
Mem_free returns mem_heap_t allocated memory and releases mem_heap_t
Mem_heap_alloc allocating a piece of memory on the specified mem_heap_t
Mem_heap_get_heap_top get the address of the heap top block to use memory
Mem_heap_empty emptying the specified mem_heap_t
Mem_heap_get_top gets the mem_block_t pointer of the specified n size at the top of the heap
Mem_heap_free_top release heap N-size mem_block_t block at top

4 Summary InnoDB provides a memory pool and heap allocation method to manage memory uniformly, the most basic purpose is to increase the memory rate.

In the version number of the MySQL-5.6. InnoDB provides two options, one is to use the memory pool provided by InnoDB to manage memory, and the other is to provide the system malloc and free as memory management. MySQL defaults to the way the system manages memory, and some experienced DBAs use the system's managed memory mode +tmalloc to do memory optimizations. Achieve improved MySQL performance with tmalloc efficient memory management.

MySQL series: InnoDB source code Analysis of memory management