Linux File read/write Analysis

In this article "Linux kernel Virtual File System Analysis", we can see how files are opened and how file read/write is triggered.

When you call the read/write System for an opened file fd, f_op-> read/f_op-> write of the file structure corresponding to the file in the kernel is called.

This article will follow this path to see how the reading and writing of common disk files is achieved.

 

The Linux kernel responds to the read/write hierarchy of a block device file (from ULK3 ):

 

 

 

1. VFS: Virtual File System.

We have seen how f_op-> read/f_op-> write is called. This is what VFS does (see Linux kernel Virtual File System Analysis).

 

2. Disk Caches: high-speed cache of disks.

Cache data on the disk in the memory to accelerate file read/write. In fact, in general, read/write only deals with the cache. (Of course, there are special cases. As mentioned below .)

Read the data directly from the cache. If the data to be read is not in the cache, a disk read operation is triggered and the data on the disk is updated to the high-speed cache. write is also directly written to the cache, then you don't need to worry about it. Later, the kernel will write data back to the disk.

 

To implement such caching, inode of each file is embedded with an address_space structure and accessed through inode-> I _mapping. A radix tree is maintained in the address_space structure, and the memory pages used for disk cache are mounted on this tree. Since the disk cache is associated with the inode of the file, each process that opens the file shares the same cache.

The specific implementation details of the radix tree can be understood as an array. Each element in the array is a page, and the file content is stored in these pages in sequence.

 

Therefore, the pos of the file to be read and written can be converted to the page number to be read and written (pos is in bytes and only needs to be divided by the number of bytes on each page ).

When inode is loaded into the memory, the corresponding disk cache is empty (there is no page on the radix tree ). As files are read and written, data on the disk is loaded into the memory, and the corresponding Memory Page is mounted to the corresponding location of the radix tree.

If the file is written, the content on the corresponding page of the radix tree corresponding to the inode is updated and will not be directly written back to the disk. This is a dirty page that has been written but not updated to the disk.

The kernel thread pdflush regularly updates the dirty pages on each inode to the disk and recycles the pages on radix in a timely manner. These contents are not discussed here.

 

When the content of the file to be read and written has not been loaded into the corresponding radix tree, the read/write execution process will initiate a read request to the underlying "general block layer" to read data.

If the O_DIRECT option is specified when the file is opened, it means to bypass the high-speed disk cache and directly deal with the "general block layer.

Since the disk high-speed cache provides a cache mechanism that is conducive to improving read/write efficiency, why should we use the O_DIRECT option to bypass it? In general, such applications maintain a dedicated cache mechanism in user mode, which is more conducive to application use, it is used to replace the General cache mechanism of the disk high-speed cache provided by the kernel. (This is usually the case for database programs .)

Since the O_DIRECT option is used, the File Cache is changed from the disk cache provided by the kernel to the user cache, different processes that open the same file will not be able to share the cache (unless these processes create a shared memory or something ). If some processes use the O_DIRECT option for the same file, but some do not? When the O_DIRECT option is not used to read and write this file, the corresponding content will be left in the disk cache; when the O_DIRECT option is used to read and write this file, you need to write the dirty data corresponding to this read and write in the disk cache back to the disk, and then directly read and write the disk.

The specific implementation details of direct_IO brought by the O_DIRECT option are not described here. See linux asynchronous IO analysis.

 

3. Generic Block Layer: general Block Layer.

The Linux kernel abstracts a unified model for Block devices and regards Block devices as an array space composed of several sectors. A sector is the smallest unit of disk device read/write. You can specify the disk sector to be accessed by the sector number.

The upper-layer read/write requests are constructed into one or more bio structures in the general block layer. This structure describes a request-The start sector ID of the access? How many sectors are accessed? Is it read or write? What are the corresponding memory pages, page offset, and data length? Wait ......

 

There are two main problems: where does the fan area number come from? How is the memory organized?

As mentioned above, the upper-layer read/write requests can locate the pages on which the corresponding disk cache is to be accessed through the file pos, on the index page, you can know the first sector of the file to be accessed and obtain the index of the sector.

However, the number of slices of a file is not the same as the number of slice on the disk. The resulting slice index needs to be converted to the sector ID by the function provided by the specific file system. The file system records the usage of the sectors on the current disk, and which sectors are used in sequence for each inode. (See linux File System Implementation Analysis)

Therefore, through the specific functions provided by the file system, the file pos requested by the upper layer is eventually mapped to the fan area number on the disk.

It can be seen that a request at the upper layer may span multiple sectors and may form multiple discontinuous sector segments. Corresponding to each sector segment, a bio structure is constructed. Because Block devices generally support one-time access to several consecutive sectors, a single sector segment (more than one sector) can be included in a bio structure representing a block device IO request.

 

Next we will talk about the memory organization. Since a read/write request on the upper layer may span multiple sectors, it may also span multiple pages cached on the disk. Therefore, the slice Request contained in a bio may correspond to a group of memory pages. These pages are allocated separately, and the memory address may be discontinuous.

So, since bio describes a block device request, can block devices access a group of consecutive slices at a time, but can they access a group of non-contiguous memory addresses at a time?

Block devices generally use DMA, copy data from a group of consecutive slices on a block device to a group of consecutive memory pages (or copy data from a group of consecutive memory pages to a group of consecutive blocks on the block device) sector ), DMA itself generally does not support one-time access to non-contiguous memory pages.

However, some architectures contain io-mmu. Just as mmu can map a set of non-consecutive physical pages to a continuous virtual address, I/O-mmu is programmed, this allows DMA to regard a set of discontinuous physical memory as continuous. Therefore, even if a bio contains non-continuous multi-segment memory, it may be completed in one DMA. Of course, not all architectures support io-mmu, so a bio may also be split into multiple device requests in the device drivers that follow.

 

Each constructed bio structure is submitted to the underlying IO scheduler.

 

4. I/O SchedulerLayer and I/O scheduler.

We know that the disk reads and writes data through the head, and the head needs to be moved mechanically when locating the sector. Compared with the transmission of electricity and magnetism, the mechanical movement is very slow, which is the main reason why the disk is so slow.

The IO scheduler must move the head as little as possible to improve the disk read/write efficiency while completing the existing requests. The most famous is the elevator algorithm ".

In the IO scheduler, bio submitted by the upper layer is constructed into a request structure, and a request structure contains a group of sequential bio. Each physical device corresponds to a request_queue, which stores related requests in sequence.

The new bio may be merged into the existing request structure in request_queue (or even merged into the existing bio), or a new request structure may be generated and inserted to the appropriate location of request_queue. How to merge and insert depends on the IO scheduling algorithm selected by the device driver. In general, I/O scheduling algorithms can be imagined as "elevator algorithms", although the actual I/O scheduling algorithms have been improved.

In addition to the I/O Scheduling Algorithm Similar to the "elevator algorithm", there is also the "none" algorithm, which actually does not have an algorithm, or it can be said to be "first come first served algorithm ". Because many devices now support Random Access (such as solid state disks and flash), using the "elevator algorithm" makes no sense for them.

 

In addition to changing the request sequence, the IO scheduler may delay request processing. Because the "elevator algorithm" can be used only when the request queue has a certain number of requests, otherwise it will degrade to "first come first served algorithm" in extreme cases ".

This is achieved through plug/unplug of request_queue. plug is equivalent to disabling and unplug is equivalent to restoring. When there are few requests, the request_queue will be disabled. When the number of requests reaches a certain level, or the "Oldest" request in request_queue has been waiting for a long time, the request_queue will be restored.

When request_queue is restored, the callback function provided by the driver is called, so the driver starts to process request_queue.

In general, the read/write system returns the result after calling it here. After the result is returned, it may wait (synchronous) or continue to do other tasks (asynchronous ). Before the return, a task is added to the task queue, and the kernel thread that processes the task queue will execute the request_queue unplug operation in the future to trigger the driver to process the request.

 

5. Device Driver.

Here, the device driver needs to retrieve requests from request_queue and then operate the hardware devices to execute these requests one by one.

 

In addition to processing requests, the device driver must select an I/O Scheduling Algorithm because the device driver knows the properties of the device and what I/O scheduling algorithm is most suitable. Even the device driver can block the IO scheduler and directly process bio on the upper layer. (Of course, the device driver can also implement its own IO scheduling algorithm .)

I/O scheduler is a set of methods provided by the kernel to the device driver. The options are device drivers.

 

Therefore, for Block devices that support random access, in addition to selecting the "none" algorithm, the driver also has a more direct approach, that is, registering its own bio commit function. In this way, after bio is generated, it will not use the generic commit function and be submitted to the IO scheduler, but directly processed by the driver.

However, if the device is slow, bio submission may be blocked for a long time. This method is generally used by memory-based Block devices (of course, such Block devices are virtualized by the driver ).

 

 

The following describes the execution process of read/write:

Sys_read. Get the corresponding file structure through fd, and then call vfs_read;

Vfs_read. Check various permissions and file locks, and then call file-> f_op-> read (if not, call do_sync_read ). File-> f_op comes from the corresponding inode-> I _fop, while inode-> I _fop is assigned when the corresponding file system type generates this inode. File-> f_op-> read is probably equivalent to do_sync_read;

Do_sync_read. F_op-> read is a synchronous read, while f_op-> aio_read completes an asynchronous read. Do_sync_read uses the f_op-> aio_read asynchronous read operation to complete synchronous read. That is, after an asynchronous read is initiated, if the returned value is-EIOCBQUEUED, the process goes to sleep, you can do so until the read is complete. However, for Disk File Reading, f_op-> aio_read generally does not return-EIOCBQUEUED unless the O_DIRECT flag aio_read is set, or for some special file systems (such as network file systems such as nfs );

F_op-> aio_read. This function is usually implemented by generic_file_aio_read or its encapsulation;

Generic_file_aio_read. One asynchronous read may contain multiple read Operations (corresponding to the readv System Call). For each of these operations, do_generic_file_read is called;

Do_generic_file_read. The main process is to find the corresponding page in the radix tree and the page is available. Yes, the required data is read from the page and then returned. Otherwise, the page is read through file-> f_mapping-> a_ops-> readpage. (File-> f_mapping-> a_ops-> readpage, the read request has been submitted. However, the data on the disk may not have been read. You need to wait for the data to be read. The waiting method is lock_page. Before calling file-> f_mapping-> a_ops-> readpage, The PG_locked mark is assigned to the page. After the data is read, the mark is cleared, which will be viewed later. The lock_page here is to wait until the PG_locked mark is cleared .);

File-> f_mapping comes from the corresponding inode-> I _mapping, and inode-> I _mapping-> a_ops is assigned by the corresponding file system type when the inode is generated. The a_ops-> readpage functions provided by various file system types are generally encapsulated by the mpage_readpage function;

Mpage_readpage. Call do_mpage_readpage to construct a bio, and then call mpage_bio_submit to submit it;

Do_mpage_readpage. Based on page-> index, determine the disk sector number to be read, and then construct a group of bio. The get_block function provided by the file system type needs to be used to correspond to the disk sector number to be read;

Mpage_bio_submit. Set bio-> bi_end_io to mpage_end_io_read and call submit_bio to submit this bio group;

Submit_bio. Call generic_make_request to submit bio to the disk drive maintenance request queue;

Generic_make_request. A packaging function calls _ generic_make_request for each bio;

_ Generic_make_request. Obtain the Request queue bio-> bi_bdev-> bd_disk-> queue of the disk object corresponding to the block device file of bio, and call q-> make_request_fn to add bio to the queue;

Q-> make_request_fn. The device driver initializes the request_queue structure during initialization, and sets q-> make_request_fn and q-> request_fn (this will be used below ). The former is used to assemble a bio into a request and add it to request_queue. The latter is used to process requests in request_queue. Generally, the device driver calls blk_init_queue to initialize request_queue. q-> request_fn must be specified, while q-> make_request_fn uses the default _ make_request;

_ Make_request. Based on different scheduling algorithms, how to add bio, generate the corresponding request structure, and add it to the request_queue structure, and decide whether to call q-> request_fn, or add a task in the kblockd_workqueue task queue and wait for the kblockd kernel thread to call q-> request_fn;

Q-> request_fn. A function defined by the driver is used to retrieve the request from request_queue for processing. Several requests have been sorted by the IO scheduling algorithm since bio is added to the request. The driver reads data from the actual physical device to the memory according to the description in the request structure. When the driver completes a request, it will call the end_request (or similar) function to end the request;

End_request. The final request is completed, and the corresponding bio end method bio-> bi_end_io will be called, that is, the mpage_end_io_read previously set;

Mpage_end_io_read. If the page has been updated, set the up-to-date tag, unlock the page, and wake up the process waiting for the page to be unlocked. Finally, the bio object is released;

 

Sys_write. Like sys_read, the corresponding vfs_write, do_sync_write, f_op-> aio_write, and generic_file_aio_write are called sequentially;

Generic_file_aio_write. Call _ generic_file_aio_write_nolock to write data to the disk cache. After writing, determine if the file is opened with the O_SYNC mark, then call sync_page_range to synchronize the data written to the disk cache to the disk (only the file header information is synchronized );

_ Generic_file_aio_write_nolock. After some checks, call generic_file_buffered_write;

Generic_file_buffered_write. Call generic_cmdm_write to execute the write. After the write is complete, it is determined that if the file is opened with the O_SYNC mark, then, call generic_osync_inode to synchronize the data written to the disk cache to the disk (synchronize the file header information and file content );

Generic_cmdm_write. An asynchronous write operation may contain multiple write operations (corresponding to the writev System Call, call file-> f_mapping-> a_ops-> write_begin to prepare the disk cache page to be written, and then merge the data to be written into the page, finally, call file-> f_mapping-> a_ops-> write_end to complete writing;

File-> f_mapping comes from the corresponding inode-> I _mapping, and inode-> I _mapping-> a_ops is assigned by the corresponding file system type when the inode is generated. The file-> f_mapping-> a_ops-> write_begin function provided by each file system type is generally the encapsulation of block_write_begin function, and the file-> f_mapping-> a_ops-> write_end function is generally the generic_write_end function. encapsulation;

Block_write_begin. Call grab_cache_page_write_begin to find the page to be written in the radix tree. If the page does not exist, create one. Call _ block_prepare_write to prepare a set of buffer_head structures for the page to describe the data blocks that constitute the page (the corresponding bio structure can be generated using the information );

Generic_write_end. Call block_write_end to submit the write request and set the dirty mark of the page;

Block_write_end. Call _ block_commit_write to set the dirty flag for each buffer_head structure in the page;

So far, the write call will return. If the file is opened with the O_SYNC mark, sync_page_range or generic_osync_inode will be called. Otherwise, write ends. Wait until the pdflush kernel thread finds the dirty pages on the radix tree and finally calls do_writepages to write back these dirty pages;

Sync_page_range is also implemented by calling generic_osync_inode, while generic_osync_inode will eventually call do_writepages;

Do_writepages. Call inode-> I _mapping-> a_ops-> writepages, which is generally the packaging of the mpage_writepages function;

Mpage_writepages. Check the page to be written back in the radix tree and call _ mpage_writepage for each page;

_ Mpage_writepage. Here we also construct bio, and then call mpage_bio_submit for submission;

The subsequent process is almost the same as read ......