InnoDB in implementing a layer of logical storage space management based on file IO, table space uses a logical hierarchical structure: space, segment Inode, Extent and page. The logic in the implementation layer uses the disk-linked list structure to manage the logical relationships. Let's start by introducing the disk list.
1. Disk Link ListImplementation of the disk list fut0lst.* file, InnoDB to manage tablespaces and index modules, defines a disk-based linked list that is primarily used to preserve the relationship between disk data structures. This list is not based on memory pointers, but rather on the location bindings based on page no and boffset. A fil_addr_t structure is defined in InnoDB to describe:
typedef struct fil_addr_struct{ulintpage; /*page number in space */ulintboffset; The byte offset in the/*page, using 2 bytes in memory to represent the */}fil_addr_t;
fil_addr_t can be fut_get_ptr function to get the memory location of node (flst_node_t)
flst_node_t can be buf_ptr_get_fsp_addr to determine fil_addr_t. The flst_node_t contains 12 bytes of content, the first 6 bytes (Page:4 boffset:2) represent the fil_addr_t information relative to their previous node, and the last 6 bytes represent the fil_addr_t of 1 node relative to themselves. In addition to flst_node_t, the disk list also has a header information flst_base_node_t, the header information is a node number Flst_len (4 bytes) + flst_first (6 bytes) + flst_ Last (6 bytes). 1.1 Structure relationship of a disk-linked list
2.space Structural AnalysisIn the InnoDB tablespace, all data is stored in page units, and there are two types in space (tablespace)page:fsp_hdr/xdes page, fseg inodes page. Each page is stored in the default 16KB size,InnoDB assigns a page a total of 64 page at a time in a extent unit. 2.1 FSP Hdr/xdes Page2.1.1xdes Structure Analysis (extent) This type of page mainly stores two types of information, the first 112 bytes are stored in the file Space header information, The remaining space behind stores multiple extent description information (xdes), the specific storage structure is as follows:
Only the first page of space saves the FSP header, and the other pages are populated with 0. Each xdes page contains a maximum of 256 xdes descritptors Entry, each xdes descritptors Entry corresponds to a extent. The structure of the Xdes descritptors entry is described below:
The File Segment ID is the ID of the Segment that the current extent belongs to
Xdes list is a node of the disk bidirectional list that points to the page position of the previous Xdes entry and the next X Page position of DES entry
State extent, Xdes_free, Xdes_free_frag, Xdes_full_frag, Xdes_fseg, when it comes to xdes_fseg, indicates that the extent has been attached to a segment,extent at the time of creation Specifies the XDES_FSEG state. A extent in the state xdes_free when it was just allocated.
Bitmap the status index of all page of the current extent, a page occupies 2 bit, the first bit indicates whether it is used State, the second bit indicates whether or not the state is cleared, and the empty state is temporarily not used, and is true.
2.1.2 FSP Header Space ID The ID of the current tablespace size The maximum number of page spaces currently available for space to change when the file is enlarged
LimitThe current space has been assigned the initialized page number, including the idle and already used
Flag Not working
Frage usedThe number of page (Fsp_free_frag) that have been used in the list
Free list List of extent objects available in space, extent not a page is used
Frag Free List There is a list of extent that can be broken leaf page, some page in exntent is used
Frag full listThere is no extent list of available page, exntent all page is used
Segment ID The next available segment ID
Full Inode ListSpace is currently fully occupied segment Inode page list
Free Inode ListSpace is currently fully occupied segment Inode page list
2.2 Fseg inode Page This page type is the page that stores the Fseg inode, each inode occupies 192 bytes, and a page stores 85 Inode objects, structured as follows:
After the FIL header is followed by 12 bytes, this 12 byte is actually a list in the full inode list or the free Inode list So, respectively, represents the front and back fil_addr_t. Each inode information occupies 192 bytes, each of which manages the corresponding extent and fragment page. The inode structure is as follows:
FSEG ID Segment ID
Not full used page number in fseg_not_full list
Fseg_free List of idle extent in Inode
Fseg_not_full extent some page is occupied, there are some page idle extent list
Fseg_full fully occupied list of extent
Fseg_magic_n Check magic Word
Fragment array a random page index with a length of 32 stored, if the data is full. The main function is Save space, for example, when a table is first established, it is not assigned a full extent to the table, only 6 page pages are allocated, this time You need to use the fragment array to manage it.
3.space structure diagram 3.1space frame diagram
3.2 Module Relationship
4.space Inode, extent, and page allocation process InnoDB space, the relationship between the inode, extent, and page is interlocking, the inode corresponds to the segment,extent corresponding to the area, page is the pages, is also the smallest allocation unit for the table space. A page in MySQL default is 16KB size, a extent Management 64 page, the size of 1M, and the inode can manage a lot of extent plus 32 Frag page (Broken page). Frag page is defined for space-saving purposes. After understanding the above basic concepts, we begin to analyze the allocation of inode, extent allocation and page allocation process.
4.1 inode allocation process through the Inode page introduction we can know that the inode information must be stored in the Inode page, when allocating the inode, must be from the Inode page to obtain the idle inode. If there is no Inode page to use, you will first get an inode page in Space's free list (in function Fsp_alloc_seg_inode_page) and then get the idle inode in this inode page. In this process will involve two disk linked lists: Fsp_seg_inodes_free and Fsp_seg_inodes_full, these two queues are management Inode page, if there is no idle inode the Inode page is placed in Fsp_seg_ In Inodes_full, if there is an inode page for the free inode, it is placed in the Fsp_seg_inodes_free. An Inode page consists of 85 inode information. The following are inode allocations:
1th step: When Fsp_seg_inodes_free is empty, get an inode page to the Space default header page, corresponding function fsp_alloc_seg_inode_page
2nd step: When applying for the inode, if Fsp_seg_inodes_free has an inode page, from the Inode page or an inode, the corresponding function Fsp_alloc_seg_inode
3rd step: If after the inode is in the Inode page has no idle inode, the Inode page will be put into fsp_seg_inode_full and removed from the fsp_seg_inodes_free.
4th step: If all the pages managed by the Inode are idle, the Inode state will be set to idle, which will move the Inode page from Fsp_seg_inode_full to Fsp_seg_inodes_free; This process is only called when the segment is deleted. The corresponding function Fsp_free_seg_inode
The allocation process of the 4.2extent extent is divided into two types, one is the application allocation through the Inode, and the other is the application of the distribution through the fragment fragmentation method. Inode allocation method is when there is no idle extent in the inode, the space free list will be applied to 1 or 5 extent for management, if the Inode management extent number less than 40 o'clock, each time only 1 extent will be requested, If you exceed this size, you will be applying for 5 extent at a time, which involves the fseg_free, Fseg_not_full, and Fseg_full of the inode with three disk lists. The second way to apply is to assign Frag page, which is to apply directly to extent, which involves both the Fsp_free_frag and Fsp_full_frag of the two disk linked lists. Here is the assignment:
, 1~7 is part of the Inode application allocation process, 8~12 is a Frag page application extent way
1: When the Inode's free list is empty, if you need to use the new Extent,innodb, you will get the free extent from the space to list and add it to the inode.
2: When there is a extent in the Inode free list, if you apply for a new extent, you only need to take it from the Inode free list and move the extent to the inode not full.
3: Only through the inode way to apply for a page operation, this time extent have enough free page.
4: When there is no idle page in the extent, the extent is transferred from the Inode not to the inode full.
5: When a page is released, the page extent is a fully occupied and inode-managed extent, then the page is released, it will move the extent from Inode full to inode not full
6: When a page is released, the page is located in the extent and only this page is occupied, then the page is released, the extent will be returned to the Inode list. And it will go directly to 7 to return the extent to the space free list.
8~12 is similar to the above steps
4.3page Allocation Process page application assignment is based on the Inode application and the extent application, the pages of the application are external through the Inode method, but also through the Fragment page method application. Fragment method application is relatively simple, it is not stated, the source code is very clear. Inode allocation is more complex, and its main implementation is in the Fseg_alloc_free_page_low and Fseg_free_page_low of the two functions. In the Fseg_alloc_free_page_low function, 7 cases were achieved to obtain the page in the Inode.
1. The page that specifies the hint location of the inode is idle and returns the corresponding pages directly
2.DESCR is idle, but the number of idle pages in the segment inode is < 1/8, and the fragment page is fully exhausted, assigning a extent to it, and getting hint corresponding page
3. If the DESCR is not idle and the number of idle pages in the segment inode is < 1/8, an idle extent is obtained in the inode, and the extent DESCR corresponding page is returned.
4.DESCR is a xdes_fseg state, and there is an idle page in this extent, from which a page is obtained.
5. In addition to the above, if the DESCR is not idle, but the inode has other idle extent, get an idle from the other extent.
6. If the other extent do not have free pages, but the fragment array has an idle fragment page, a free page is obtained from the free fragment page.
7. If there is no broken page, apply directly to assign a new extent and get a free page in it.
5. OverviewThe implementation of table space is also dependent on files such as page0page.* fil0fil.* in the fsp0fsp.* file. InnoDB on storage, the smallest storage unit is defined as Page,space is designing these layer relationships for more efficient and reasonable management of the page. Space can exist in the same database file as other tables, or it can be stored in a single table. This depends on the configuration of MySQL. Analyzing the structure and workings of space helps us understand how InnoDB is stored, and it provides a strong foundation behind understanding indexes, locks, and transactions. It also says that the smallest unit of storage is page, and I'll describe how the data page is stored and how it works in the next section.
MySQL Series: InnoDB source analysis Table space management
