Full analysis of the lru module of the mysql kernel source code deep Parsing Buffer Pool (bufferpool Part 2)

Full analysis of the lru module of the mysql kernel source code deep Parsing Buffer Pool (bufferpool Part 2)

Liu's original article, CSDN first! Indicate the source for reprinting.

LRU module components

(1) Overall LRU Operating Mechanism

To fully understand the bufpool subsystem, we must break through modules one by one. In my current experience, LRU-> flush-> buf read-> buddy allocator-> buf pool is recommended.

The first is the relationship diagram of the five modules,In general, LRU is not only an algorithm in the system, but also a linked list and an important component module. The author's sentence "These statistics are not 'of 'lru but' for 'lru." is the best interpretation.

The management part of the buf pool instance calls the most LRU function interfaces, and The read cache, flush, and partner systems also have important calls.

In addition,Storage subsystemThe fil0fil. c file in the file tablespace Management Section also references one of the LRU function interfaces.

 

(2) LRU linked list structure and basic algorithms

The structure of the LRU linked list is simple and direct. It is divided into the old part and the header part, which involves a policy:

The buf_pool_struct struct hasLRU_oldObject, which is a pointer of the buf_page_struct type pointing to the "old part" of the LRU linked list.

Buf_page_t * LRU_old;

Another object,

Ulint LRU_old_ratio;

Using the macro BUF_LRU_OLD_RATIO_DIV to define the LRU linked list's OLD-end pointer setting and the old-end length of the buf pool is also an important object.

When the LRU linked list length exceeds a threshold value, the LRU_old object of the struct is initialized, that is, the following logic:

UT_LIST_GET_LEN (buf_pool-> LRU) = BUF_LRU_OLD_MIN_LEN

The OLD part of the LRU linked list involves four macro definitions:

The first one indicates the number of nodes with the LRU length, and then the LRU_old initialization operation is triggered. The length is 512;

# Define BUF_LRU_OLD_MIN_LEN 512/* 8 megabytes of 16 k pages */

The second represents the denominator of the buf_pool_struct struct object LRU_old_ratio, which is an important definition for determining the position of the LRU_old pointer.

# Deprecision BUF_LRU_OLD_RATIO_DIV 1024

Old lower limit

# Define BUF_LRU_OLD_RATIO_MIN 51

Old limit

# Define BUF_LRU_OLD_RATIO_MAX BUF_LRU_OLD_RATIO_DIV

 

Now, combine and sort the above information,When the number of nodes in the LRU linked list is lower than the number of BUF_LRU_OLD_MIN_LEN, the old end of the LRU linked list does not need to be initialized. buf_pool_t-> LRU_old = NULL;

The old object Value of the buf_page_struct struct is also false.

The above macros are defined at the beginning, and LRU_old_ratio is completed during the initialization of the buf pool, which is called by the buf_pool_init () function:

Buf_LRU_old_ratio_update (100*3/8, FALSE );

It is very clear that the default value of (LRU_old_ratio/BUF_LRU_OLD_RATIO_DIV) is 3/8.

The buf_LRU_old_ratio_update function calculates lru_old_ratio_div Based on the preset values of BUF_LRU_OLD_RATIO_DIV and 3/8.Related code:

......

If (ratio <BUF_LRU_OLD_RATIO_MIN ){

Ratio = BUF_LRU_OLD_RATIO_MIN;

} Else if (ratio> BUF_LRU_OLD_RATIO_MAX ){

Ratio = BUF_LRU_OLD_RATIO_MAX;

}

......

If the upper limit BUF_LRU_OLD_RATIO_MAX is greater than 3/8, LRU_old_ratio is assigned according to the upper limit. If the value is smaller than the lower limit BUF_LRU_OLD_RATIO_MIN, it is set to the lower limit, make the length of the old end flexible (that is, the length of the old end is not always 3/8 after the length of the linked list is adjusted), and can be used for flush or remove.

(3) Function Classification

The main body of the source code is 43 functions. Function interfaces are classified from the functional point of view and the calling relationships are sorted out. Basically, these functions focus on three aspects. At the beginning, they were summarized as follows:

15 important main functions:The basic functions are centered on allocating blocks, releasing blocks, deleting blocks from the LRU linked list, and adding blocks to the LRU linked list, however, as the flush mechanism, free idle linked list, old-end control of LRU linked list, and virtual memory disk replacement of bufpool are involved, up to 15 main function interfaces are derived for external modules to call. In addition, the so-called 'main function' is not accurate, but it is a fuzzy positioning based on the number of calls and the importance of the function. Many of the auxiliary functions are not called externally, but in a sense, it is even more important!

14 auxiliary functions closely related to the main function:Many of these 14 Functions contain the implementation details of the first class of main functions, and are the essence of flesh and blood.

Other function interfaces: minus 43 Functions15 main functions and other functions except 14 auxiliary interfaces are not listed here.

The table column is as follows, with no ranking in any order:

Type	Function	Name
Main Function	Delete all pages of a given tablespace (upper layer interface)	Buf_LRU_flush_or_remove_pages
	Allocate idle blocks (upper Interface)	Buf_LRU_get_free_block
	Add a block to the LRU linked list (upper Interface)	Buf_LRU_add_block
	Release blocks, delete blocks from LRU, and add them to the freeL linked list.	Buf_LRU_free_block
	Release blocks, delete blocks from LRU, and add freeL linked list (upper Interface)	Buf_LRU_search_and_free_block
	Obtain a block from a free idle linked list	Buf_LRU_get_free_only
	Delete the refreshed block from the LRU linked list and add it to the free linked list.	Buf_LRU_try_free_flushed_blocks
	Place the block back to the free linked list	Buf_LRU_block_free_non_file_page
	Delete a block from the LRU linked list and hash table	Buf_LRU_block_remove_hashed_page
	Release blocks from the LRU linked list and add them to the free linked list (upper Interface)	Buf_LRU_free_one_page
	LRU core old parameter update	Buf_LRU_old_ratio_update
	Add the block to the LRU Header	Buf_LRU_make_block_young
	Place the block at the end of LRU	Buf_LRU_make_block_old
	Dump LRU page to disk	Buf_LRU_file_dump
	Read the LRU page from the dump file	Buf_LRU_file_restore
Associate auxiliary functions	Deletes the LRU linked list node on the memory page of a given table in all buf pools.	Buf_LRU_remove_all_pages
	Delete or refresh the dirty page of the given tablespace in the bufpool (upper layer interface)	Buf_flush_dirty_pages
	Deletes the dirty page of The tablespace to which the table space belongs. It is the upper-layer loop method of the next function.	Buf_flush_or_remove_pages
	Delete the tablespace dirty page	Buf_flush_or_remove_page
		Buf_LRU_drop_page_hash_for_tablespace
	Determine whether to clear the end Of the unzip_LRU linked list	Buf_LRU_evict_from_unzip_LRU
	Release non-compressed pages from the unzip_LRU linked list	Buf_LRU_free_from_unzip_LRU_list
	Release clean pages from LRU (essentially if the flush linked list is not modified)	Buf_LRU_free_from_common_LRU_list
	Remove a block from the LRU linked list	Buf_LRU_remove_block
	If the block to be removed is also in unzip_LRU, remove it together.	Buf_unzip_LRU_remove_block_if_needed
	Add a block to the end of LRU	Buf_LRU_add_block_to_end_low
	Add a block to the LRU linked list (underlying implementation)	Buf_LRU_add_block_low
	Add file page blocks without hash indexes to the free linked list	Buf_LRU_block_free_hashed_page
	Method for updating LRU_old_ratio, an important object of the buf_pool struct	Buf_LRU_old_ratio_update_instance
Other functions	No description	Buf_LRU_old_init
		Incr_LRU_size_in_bytes
		Buf_LRU_old_adjust_len
		......

Note: 1: (upper-layer interface) indicates that the function has no implementation details or major implementation details, so that other main functions or auxiliary functions can be referenced to complete the function.

2: The author does not directly copy the comments of functions. Heikki is actually a generation of pitfalls. Some Function comments and function descriptions are not accurate in detail, I can't see the method from the brief description of the function, and even think that many functions are redundant.

From the perspective of this table, it will give people a dazzling feeling, it is difficult to switch to the main line of function calling in the LRU section. This section introduces the function call analysis section below.

 

(4) Main Line of function calling

First, let's explain that this part is only a detailed analysis of the relationship between the main line and the branch line of function calls, rather than a detailed analysis of functions. For detailed function analysis, I will select more than 10 most representative and important functions from all 43 functions after completing the overall detailed analysis of LRU, and open up a separate chapter, there will be a lot of content. In this section, we will only extract some source code based on the need to analyze the function call relationship for a very brief description.

The functions of the LRU component are complicated, and even people may feel "overlapping functions". In fact, from the perspective of calling relationships, it is not impossible to quickly recognize them through simple and direct means. This requires analysis of the source code and call of several functions:

Buf_LRU_get_free_block

Buf_LRU_search_and_free_block

Buf_LRU_block_remove_hashed_page

Buf_LRU_block_free_hashed_page

Buf_LRU_flush_or_remove_pages

Buf_LRU_add_block

Basically, these six functions cover the most important core functions of LRU.

Starting with buf_LRU_get_free_block, this function is used to allocate blocks ). The assigned action contains four main functions, three of which are from the LRU module itself,A buf_flush_free_margin from the flush Module(The function is to refresh the dirty blocks in the LRU linked list. The flush part focuses on this ).

 

The functions of the other three LRU modules show that these four functions are called in parallel within the function, rather than nested call relationships.

The first call to buf_LRU_get_free_only. The initial action for allocating blcok is to search from the free linked list. If the condition is met, the function returns the result directly. The function contains no more than 40 rows in the buf0lru. c 1,155th starts.

The second call to buf_LRU_search_and_free_block is much more complicated, and the descriptions of functions should be sufficient. But we need to emphasize two points. The first point is that the allocation block must be from the free linked list. Even if the lru linked list is cleaned up, finally, you need to add the cleaned block to free. Second, remove the linked list from lru or non-dirty pages (the so-called "clean LRU linked list node ") perform "evict" according to the policy. If dirty pages have been refreshed in the flush linked list, they can be removed. Pay special attention to these two points.

The third call is a two-in-one function. buf_flush_free_margin refreshes the LRU dirty pages (as described in the second point above). Dirty pages that fail to be refreshed must be locked in LRU, you must wait for the synchronous or wake-up asynchronous IO thread to complete the fsync action. For the other function buf_LRU_try_free_flushed_blocks, it can be seen that the function is identical to the buf_LRU_search_and_free_block function and belongs to the upper Interface.

The main call analysis of the buf_LRU_get_free_block function is completed,Now, we will analyze the actions of the buf_LRU_search_and_free_block function,Because of the complicated and key functions of its internal call lines.

Let's take a look at the call relationship diagram of the buf_LRU_search_and_free_block function.

 

The function first judges two branches. buf_LRU_free_from_unzip_LRU_list is complete.Non-compressed pages and compressed pages(The function name of Heikki Tuuri cannot be used to release the unzip_LRU and LRU linked lists at the same time. buf_LRU_free_from_common_LRU_list only releases the compressed page, that is, only the LRU linked list.

After figuring out this branch, let's look at the nested callThe lower-level function buf_LRU_free_block is the core call function that releases the LRU block and adds it to the free linked list!The figure shows two steps,The first step is to remove the LRU linked list by using the buf_LRU_block_remove_hashed_page function,This function contains many detailed actions and encapsulation functions for releasing LRU nodes, which will be analyzed below;

Step 2: add the free linked list and use the buf_LRU_block_free_hashed_page function.Complete. The two actions are Between 1897 and 2059 rows of the buf0lru. c.

......

If (buf_LRU_block_remove_hashed_page (bpage, zip)

! = BUF_BLOCK_ZIP_FREE ){

Ut_a (bpage-> buf_fix_count = 0 );

......

If (have_LRU_mutex)

Mutex_enter (& buf_pool-> LRU_list_mutex );

Mutex_enter (block_mutex );

If (B ){

Mutex_enter (& buf_pool-> zip_mutex );

Buf_page_unset_sticky (B );

Mutex_exit (& buf_pool-> zip_mutex );

}

Buf_LRU_block_free_hashed_page (buf_block_t *) bpage, FALSE );

Let's take a look at the call relationship diagram of the buf_LRU_block_remove_hashed_page function,How to remove the LRU linked list node.

Friends who are not dizzy here should be lucky and hard-working. Of course, it may also indicate that Liu's plotting level may be very general :), in general, it should be considered that the deletion operation of LRU nodes is relatively simple, but in the bufpool subsystem of mysql, to balance access efficiency, capacity limit, timeliness, and other factors, the LRU management mechanism must take many measures. The Node information in the hash table is used to accelerate the positioning of the block. In addition, the unzip_LRU is a subset of the LRU linked list. When removing a block from the LRU linked list, you must determine the block (buf_page_t) in this function) whether the corresponding non-compressed page control block (buf_block_t) is also in the unzip_LRU linked list. This is the six chain tables mentioned above, the five linked lists (free, LRU, flush, zip_clean, zip_free []) are all buf_page_t, but buf_block_t is in unzip_LRU.

 

Let's look back at the functions,1. Clear the LRU node and 2. Clear the hash table (a function can be used directly ),The first step is to remove the LRU node and the unizp_LRU node,Whether to remove the unzip_LRU node is completed in the handler function. The action is called by the buf_page_belongs_to_unzip_LRU function in the next layer. The buf_page_belongs_to_unzip_LRU function is in the buf0buf. ic,Determine the data Object of the buf_page_t struct object zip, the status of the buf_page_t, and the two elements;

Return (bpage-> zip. data &&

Buf_page_get_state (bpage) = BUF_BLOCK_FILE_PAGE );

When the buf_page_t state is BUF_BLOCK_FILE_PAGE and the compressed page data has been loaded into the LRU linked list of the bufpool, it must exist in the unzip_LRU linked list.

This introduces another enumeration structure and the zip object in the buf_page_struct structure.

Enumbuf_page_state {

......

BUF_BLOCK_ZIP_DIRTY ,/*! <Contains a compressed

Page that isin

Buf_pool-> flush_list */

BUF_BLOCK_FILE_PAGE ,/*! <Contains a buffered file page */

......

}; // First, list two items. BUF_BLOCK_ZIP_DIRTY is in the flush linked list. Generally, BUF_BLOCK_FILE_PAGE must be in the unzip_LRU linked list.

Struct buf_page_struct {

......

Page_zip_des_t zip ;/*!

(Butnot the data it points to) is

Alsoprotected by buf_pool-> mutex;

State = BUF_BLOCK_ZIP_PAGE and

Zip. data = NULL means an active

Buf_pool-> watch */

......

}; // Zip is actually another struct reference page_zip_des_struct. This section will be described in detail in the storage file system. Here, zip. data can be viewed as the start page frame of the compressed data page. For the four main structures such as buf_page_struct and all other important data structures in the bufpool, a separate section will be opened to classify and describe each key point in detail.

Until now, the three-layer nested call Branch of the function buf_LRU_get_free_block has not been described, that is, buf_LRU_block_free_hashed_page. Let's see the relationship graph:

 

This should be the simplest layer of calls so far.

In the buf_LRU_block_free_non_file_page function, in addition to the free linked list, initialization of the input parameter buf_block_t is also performed.

// The Block status changes. The enumerated elements mentioned above are still involved.

Void * data;

Buf_block_set_state (block, BUF_BLOCK_NOT_USED );

// Non-compressed page frame Initialization

Memset (block-> frame, '\ 0', UNIV_PAGE_SIZE );

// Compress page data

Data = block-> page.zip. data;

// Release the memory of the partner system if the data is not empty

If (data ){

Block-> page.zip. data = NULL;

Mutex_exit (& block-> mutex );

// Buf_pool_mutex_exit_forbid (buf_pool );

 

Buf_buddy_free (

Buf_pool, data, page_zip_get_size (& block-> page.zip ),

Have_page_hash_mutex );

 

// Buf_pool_mutex_exit_allow (buf_pool );

Mutex_enter (& block-> mutex );

Page_zip_set_size (& block-> page.zip, 0 );

}

// Finally add the block to the linked list.

UT_LIST_ADD_FIRST (free, buf_pool-> free, (& block-> page ));

So far, a main call line buf_LRU_get_free_block,

The call relationship and outline flowchart of the three call branches have been analyzed. The following are the following:

Buf_LRU_get_free_block (called by the main line)

Buf_LRU_search_and_free_block (secondary branch, remove LRU, and add free node)

Buf_LRU_block_remove_hashed_page (level 3 branch, remove LRU)

Buf_LRU_block_free_hashed_page (third-level branch, add a free linked list)

The whole flow chart of the function is useless. If you understand it, this graph can be completed by yourself.

 

The main (Branch) line of the other two function call relationships in the LRU SectionBuf_LRU_add_block and buf_LRU_flush_or_remove_pages are much less complex than buf_LRU_get_free_block.

Shows the Call Line of buf_LRU_flush_or_remove_pages,It is divided into two basic branches: BUF_REMOVE_ALL_NO_WRITE and BUF_REMOVE_FLUSH_NO_WRITE. These two values are the objects in the enum buf_remove_t enumeration element.

Enum buf_remove_t {

BUF_REMOVE_ALL_NO_WRITE,

BUF_REMOVE_FLUSH_NO_WRITE

};

This function is fil0fil. the delete tablespace function fil_delete_tablespace in the c (file storage subsystem) section is called to clear the cache pages corresponding to all tablespaces in the bufpool or the cache pages that only exist in the flush linked list.

The BUF_REMOVE_ALL_NO_WRITE in the enumeration element shows the action to clear all cached pages.The first function buf_LRU_drop_page_hash_for_tablespace on this side is used to clear all cache page hash entries,The buf_LRU_remove_all_pages function cleans up the given tablespace pages in the LRU linked list.In fact, this function hides several details and determines whether to call the buf_flush_remove function (the flush module clearing function) based on whether the oldest_modification value of buf_page_t is 0. If oldest_modification! = 0 indicates that the flush linked list must be cleaned up after the modification is made. Otherwise, the buf_LRU_block_remove_hashed_page is called directly. This function has been analyzed in detail previously.

......

If (bpage-> oldest_modification! = 0 ){

Buf_flush_remove (bpage );

}

......

/* Remove from the LRU list .*/

If (buf_LRU_block_remove_hashed_page (bpage, TRUE)

! = BUF_BLOCK_ZIP_FREE ){

Buf_LRU_block_free_hashed_page (buf_block_t *) bpage, TRUE );

......

}

......

Let's look at the left part of the image. In the enumeration element, BUF_REMOVE_FLUSH_NO_WRITE,It indicates clearing the cache page of The tablespace corresponding to the table space to be deleted in the flush linked list. As shown in the figure, it is actually simpler than that on the right. Three single lines directly call the buf_flush_remove function of the flush module, delete the flush node.

The above analysis section contains the main details of the buf_LRU_flush_or_remove_pages function. The following describes the call line chart of buf_LRU_add_block.

Buf_LRU_add_block is the upper-layer function interface of buf_LRU_add_block_low. In fact, it only calls the buf_LRU_add_block_low function.

 

Adding blocks to the LRU linked list is not complex. It is divided into three parts. After adding the LRU linked list, set the old end based on whether the LRU linked list is> BUF_LRU_OLD_MIN_LEN (as mentioned above) adjust the length of the linked list, and finally determine whether the compressed block has decompress blocks. You need to add them to the unzip_LRU linked list.

If (! Old | (UT_LIST_GET_LEN (buf_pool-> LRU) <BUF_LRU_OLD_MIN_LEN )){

UT_LIST_ADD_FIRST (LRU, buf_pool-> LRU, bpage );

......

}

......

If (buf_page_belongs_to_unzip_LRU (bpage )){

Buf_unzip_LRU_add_block (buf_block_t *) bpage, old );

}

For details about this function, I will provide the complete version in the detailed code analysis of buf_LRU_add_block_low.

So far, the context of the LRU module should be very clear. These six function interfaces for external subsystems or modules do not represent all the focus of the LRU module, but they give priority to the main line (branch line) Structure of the six main functions, the overall understanding of the overall structure of the LRU module and the impact of its role on external modules play a crucial role in refining the design ideas of the LRU module.

The next section provides a comprehensive analysis of the FLUSH module.