[Go] Device Mapper mechanism in the Linux kernel

This paper introduces the device mapper mapping mechanism in Linux kernel with specific code. Device Mapper is a mapping framework mechanism from logical devices to physical devices provided in the Linux 2.6 kernel, in which the user can easily develop a management strategy to implement storage resources according to their own needs, and the current more popular Linux logical Volume Manager LVM2 (Linux Volume Manager 2 version), EVMS (Enterprise Volume Management System), Dmraid (Device Mapper Raid Tool) And so on is based on this mechanism to achieve. Understanding this mechanism is the basis for further analysis and understanding of the implementation and design of these volume managers. Through this paper, we can further understand the design and implementation of Linux block-level IO.

Device Mapper is a universal device mapping mechanism in the Linux2.6 kernel that supports logical volume management, which provides a highly modular kernel architecture for implementing block device drivers for storage resource management, 1.

Figure 1 Kernel architecture for Device mapper

In the kernel it is through a modular target driver plug-in implementation of the IO request filtering or redirection, such as the current implementation of the target driver plug-in includes soft raid, soft encryption, logical tape, multipath, mirroring, snapshots, etc., in the figure linear, Mirror, snapshot, and multipath represent these target driver. Device Mapper further embodies the principle of separation of policies and mechanisms in the Linux kernel design, putting all policy-related work into the user space, and the kernel provides the mechanisms needed to complete these policies. Device Mapper User space-related sections are primarily responsible for configuring specific policy and control logic, such as logical devices and which physical devices establish mappings, how to establish these mappings, and so on, and the specific filtering and redirection of IO requests is done by the relevant code in the kernel. So the entire device mapper mechanism consists of two parts-the device mapper driver for the kernel space, the device Mapper library for the user space, and the Dmsetup tool it provides. In the following sections, we describe both the kernel and the user space. Kernel part

The kernel-related code for device Mapper has been integrated into the kernel source as part of the Linux 2.6 kernel release, and its code files can be classified as implementing device Mapper in the driver/md/directory of the kernel source code. The kernel has two parts of the basic schema file and the target driver plug-in file that implements the specific mapping work. The following analysis results are mainly based on the above source files.

Important Concepts

Device mapper is registered as a block device driver in the kernel and contains three important object concepts, mapped device, mapping table, and target device. Mapped device is a logical abstraction that can be understood as a logical device that is provided out of the kernel, and it is mapped by mapping the mapping tables described and target device. A mapping table from Mapped device to a target device is represented by a multivariate group that represents the starting address, range, and address offset of the physical device on which the Mapped device logic is represented, and the target type Equal variables (these addresses and offsets are in the sector of the disk, which is 512 bytes in size). Target device represents the physical space segment mapped by mapped device, which is the physical device that the logical device is mapped to for the logical device represented by mapped device. The three objects in device mapper together with the target driver plug-in form an iterative device tree. The top-level root node in the tree structure is the mapped device that is ultimately provided as a logical device, and the leaf node is the underlying physical device represented by target device. The smallest device tree consists of a single mapped and target device. Each target device is exclusive to mapped device and can only be used by a single mapped device. A mapped device can be mapped to one or more target device, and a mapped device can be used as the target device for its upper mapped device, which can theoretically be in device Infinite iterations under the mapper architecture. Figure 2 The hierarchical relationship of the objects in the Device mapper kernel

In Figure 2 we can see that mapped Device1 has mapped the mapping table with a, B, c three target device, and target device A has evolved through mapped device 2, mapped device 2 A mapping relationship is established through the mapping table and target device D.

Let's take a closer look at the specific implementations of the above three objects in the code, the MAPPED_DEVICE structure defined by the Dm.c file is used to represent the mapped device, which mainly includes the mapped device-related locks. Registers the request queue and some memory pools, as well as pointers to the mapping table to which it corresponds. The mapping table for the Mapped device is represented by the dm_table structure defined in the dm_table.c file, which contains an array of dm_target structures, and the dm_target structure describes mapped_device to one of its targe The mapping relationship of the T device. In the dm_table structure, these dm_target are organized in the form of B-tree to facilitate the lookup operation when the IO request is mapped. The dm_target structure specifically records the start address and range of the mapped device logic region mapped by the target device, along with pointers to the target_type structure of the specific target device-related operations. The TARGET_TYPE structure contains the name of the target driver plugin for target device, defines the method for constructing and deleting this type of target device, and the target Device corresponding to the IO request remapping and end Io method, etc. The domain that represents the specific target device is the private domain in Dm_target, which points to the structure of the specific target device mapped by mapped device. The specific structure of the target device is different for different target types, such as the simplest linear mapping target type the structure of the target device is the LINEAR_C structure defined in the Dm-linear.c file. It is defined as follows:

struct Linear_c {struct dm_dev *dev;sector_t start;};

The definition of the target device is fairly straightforward and includes only the DM_DEV structure pointer that represents the corresponding physical device and the offset address start in sector units in the physical device. The above several data structures are shown in relation to 3:

Figure 3 Relationship of several important data structures in device mapper

Build process in kernel

Below is a brief introduction to the process of creating a mapped device in the kernel, with specific code:

1, according to the kernel to the user space provided by the IOCTL interface parameters, using the Dev_create function in the dm-ioctl.c file to create the corresponding mapped device structure. This process is very simple, mainly to apply for the necessary memory resources to the kernel, including mapped device and the memory pool for the IO operation Pre-request, register the mapped device corresponding request queue through the Blk_queue_make_request function provided by the kernel dm_ Request The mapped device is registered with the kernel as a disk block.

2. Call Dm_hash_insert to insert the created mapped device into a global hash table in device mapper, which holds all the mapped device currently created in the kernel.

3. The user Space command invokes the Table_load function through the IOCTL, which constructs a mapping table for the specified mapped device and a mapped target device based on the parameters of the user space. The function first constructs the corresponding dm_table, dm_target structure, and then calls the Dm_table_add_target function in dm-table.c to initialize these structures according to the parameters passed in by the user, and according to the target type specified by the parameter, Call the constructor of the corresponding target type CTR constructs the corresponding structure of the target device in memory, and then updates the B-tree maintained in the dm_table based on the DM_TARGET structure established. After the above process is complete, the established dm_table is added to the Hash_cell structure of the mapped device's global hash table.

4, finally through the IOCTL call Do_resume function to establish the binding relationship between the mapped device and the mapping table, in fact, the process is to use the Dm_swap_table function to assign the current dm_table structure pointer value mapped_ Device in the corresponding map domain, and then modify Mapped_device to represent the current state of the domain.

With the 4 main steps above, device Mapper creates a mapped device logic block in the kernel that can be supplied to the user.

IO stream

The essential function of Device Mapper is to forward the IO request from the logical device mapped device to the corresponding target device based on the mapping relationship and the IO processing rules described by target driver. Device Mapper handles all generic_make_request and Submit_bio interfaces from the block-level IO subsystem in the kernel [two interface specific descriptions can be viewed in the references [1] and [2], both of which have a more detailed explanation of the block IO layer in the kernel. ] is directed to all blocks of mapped device read/write IO requests. The IO request is processed from the top to the bottom in the device tree of the devices mapper through request forwarding. When a bio request is forwarded to the lower mapped deivce in the device tree, one or more of the bio clones are created and sent to the underlying target device. Then the same process repeats at every level of the device tree, as long as the device tree is large enough that the forwarding process can go on indefinitely. In a hierarchy on the device tree, when target driver ends a bio request, the event that ends the bio request is escalated to its upper mapped device, which is carried out at various levels until the event eventually uploads to the root mapped device. Device Mapper then ends the original bio request on the root mapped device, ending the entire IO request process.

When bio-level forwarding is carried out on device mapper, it is eventually forwarded to one or more leaf target nodes for termination. Because a bio request cannot span multiple target device (that is, the physical space segment), device mapper clones one or more bio based on the target mapping information of the mapped device that the user has told beforehand, at each level. The bio is split and forwarded to the corresponding target device. The cloned bio is processed on the corresponding target driver on the mapped device, and the IO request is filtered according to the IO processing rules defined in target driver, and then submitted to target device for completion. The above procedure is done in the Dm_request function in the dm.c file. Target driver can do the following with these bio:

1, these bio in the drive internal queue wait for later processing;

2. redirect bio to one or more target device or to a different sector on each target device;

3. Return error status to device mapper.

The IO request is processed layered through the device tree shown in Figure 2, as described above, until the IO request ends.

Summary

Device Mapper provides a mapping scheme from logical devices to physical devices in the kernel, as long as the user has a mapping strategy in user space and writes the target driver plugin to handle specific IO requests as needed. It is convenient to implement a logical volume manager similar to LVM. Device Mapper provides an external interface in the form of an IOCTL, in which the user sends an IOCTL command to device Mapper's character devices through a user-space Device Mapper library to complete inward communication. It also provides the desired event notification mechanism through the IOCTL, allowing target driver to transmit certain IO-related events to the user space. User space Section

Device mapper is relatively simple in user space, mainly including device mapper libraries and dmsetup tools. The Device Mapper Library is an encapsulation of the necessary actions for the IOCTL, user space to create the removal device mapper logical device, Dmsetup is a command-line tool that is available directly to the user to create the removal device mapper devices. Because their functions and processes are relatively simple, the details of them are not covered in this article, the user space is mainly responsible for the following work:

1, find each mapped device related target device;

2, according to the configuration information to create a mapping table;

3, the user space to build a mapping table into the kernel, so that the kernel to build the mapped device corresponding to the dm_table structure;

4. Save the current mapping information for future rebuild.

Below we mainly illustrate the use of dmsetup through examples, and further explain the device mapper this mapping mechanism. The most important work in user space is to build and save the mapping table, here are some examples of mapping tables:

1)
0 1024x768 LINEAR/DEV/SDA 204
1024x768 Linear/dev/sdb 766

1536 LINEAR/DEV/SDC 0

2) 0 2048 striped 2 64/dev/sda 1024/dev/sdb 0

3) 0 4711 Mirror Core 2 nosync 2/DEV/SDA 2048/dev/sdb 1024

Example 1 maps the logical device 0~1023 sector, 1024~1535 sector, and 1536~1663 three address ranges to the/DEV/SDA device No. 204 sector,/DEV/SDB devices, and/dev/, respectively, in a linear map The area where the No. 0 sector of the SDC device begins.

Example 2 starts the logical device from sector No. 0, the segment with a length of 2048 sectors is striped to the 1024th sector of the/DEV/SDA device, and the area where the No. 0 sector of the/DEV/SDB device begins. Also tell the kernel that this stripe type of target driver exists 2 stripe devices mapped with the logical device, and the stripe size is 64 sectors, which allows the driver to split the IO request across devices.

Example 3 starts the logical device from sector NO. 0, and segments with a length of 4,711 sectors are mirrored to the No. 2048 sector of the/DEV/SDA device and to the area where the 1024th sector of the/DEV/SDB device begins.

After the mapping table is determined, the operation of creating and deleting logical devices is relatively simple, and the corresponding operation can be done by dmsetup the following commands.

Dmsetup Create Device Name Mapping Table File/* Creates a logical device based on the specified mapping table */
Dmsetup Reload Device Name Mapping table File/* Read the mapping file from disk for the specified device and rebuild the mapping relationship */
Dmsetup Remove Device name/* Delete the specified logical device */Figure 4 The logical device established in the kernel by the mapping table in Example 1

When user space creates a logical device command based on the mapping table, device Mapper establishes a mapping of the logical address to the physical address in the kernel based on the parameters passed in and the mapping relationship. Based on the mapping in example 1 of the mapping table set up in Device 4, the lower half of the diagram depicts the mapping of the logical address to the physical address established in the kernel by the mapping table.

The user space portion of device mapper is optional for developers to implement their own storage management tools, in fact, many of our common logical volume managers, such as LVM2, Dmraid, and other tools use device Mapper's supplied device Mapper User space Library, according to their own management needs to establish a separate set of management tools, and do not use the Dmsetup tools it provides, even IBM's Open source project enterprise-level logical Volume management system-EVMS, in the implementation of the device mapper User space library is not used, A set of its own library of functions is implemented completely according to the IOCTL definition in the kernel. Target Driver

Device Mapper provides a unified architecture that allows users to specify their own IO processing rules based on actual needs via the target driver plugin, so target driver fully embodies the flexibility of device mapper. We have mentioned target driver more than once, and we have described the function of target driver, where we combine the simplest linear target driver to specifically describe the implementation of target driver.

Target driver primarily defines the processing rules for IO requests, and the operation of the target driver in device Mapper has a unified interface defined in the implementation, which is defined in the TARGET_TYPE structure we mentioned earlier, It defines the following methods for the target driver:

1, the method of constructing target device;
2, delete the target device method;
3, Target's mapping Io request method;
4. Target end Io request method;
5, suspend the target device reading and writing methods;
6, restore the target device read and write access;
7, access to the current target device status;
8, Target processing user message method;

Users can selectively implement the above method according to the specific needs, but generally at least to achieve the first 3 methods, otherwise under the device mapper can not work properly. Linear target driver only implemented the first 3 methods and method 7, which completes the logical address space to the physical address space of the linear mapping, can be a number of physical devices in a linear connection to form a logical device, as described in 4, through the linear target Driver a large logical block device with three contiguous spaces of/DEV/SDA,/dev/sdb, and/DEV/SDC. The implementation of Linear target is simple, and its creation and deletion methods mainly complete the application and release of memory resources describing the structure used by the Linear target device; The implementation of the IO map processing method is simple, as shown in the following code:

static int Linear_map (struct dm_target *ti, struct bio *bio,      Union map_info *map_context) {struct Linear_c *LC = (struc T linear_c *) Ti->private;bio->bi_bdev = Lc->dev->bdev;bio->bi_sector = Lc->start + (bio->bi_ Sector-ti->begin); return 1;}

The mapping method is a bio request that is sent to the logical device mapped device, redirected linearly to the corresponding location of the physical device represented by the linear target device according to the mapping relationship, as shown in the code to modify the bio's Bi_ The BDEV device pointer is the device pointer for target device, and the redirect of the IO request is completed by changing the sector area code Bi_sector where the IO request begins, based on the start address of the target device and the offset value on the bio request on the mapped device devices. Other target driver implementations are similar, according to the interface specifications defined by device Mapper, combined with their own needs of the implementation can be, here is not introduced, interested readers can see the kernel of the specific target driver code.

Summarize

Device Mapper is a primary mapping mechanism provided at the block level in the Linux operating system and is now used by the logical Volume Manager under most Linux. Under this mechanism, it is very convenient to implement user-defined storage resource management strategy. Understanding the mapping mechanism provided by device Mapper is also the basis for further understanding of some common logical Volume manager implementations under Linux. Resources

1. Daniel P. Bovet, Marco Cesati. Understanding the Linux Kernel, 3rd Edition. O ' Reilly, 2005
2. Jonathan Corbet, Alessandro Rubini, Greg Kroah-hartman. Linux Device Driver, 3rd Edition. O ' Reilly, 2005
3. Linux-2.6.15 kernel source code
4. device-mapper.1.02.05 code
5. http://sourceware.org/dm/
6. Heinz mauelshagen dmraid-device-mapper RAID tool. Proceedings of the Linux Symposium 2005
7. E.goggin, A.kergon, C.varoqui, &d.olien. Linux multipathing. Proceedings of the Linux Symposium 2005