Linux IO scheduling and LinuxIO Scheduling

Linux IO scheduling and LinuxIO Scheduling
Linux I/O Scheduling

IO scheduling occurs at the I/O scheduling layer of the Linux kernel. This layer is for the overall I/O hierarchy of Linux. From the perspective of read () or write () system calls, the overall Linux IO system can be divided into seven layers, which are:

There is a compiled Linux IO structure, which is very classic. A figure is better than a thousand words:

What we want to study today is mainly at the I/O scheduling layer. How can I improve the overall performance of block device I/O? This layer is mainly designed for the mechanical hard drive structure. As we all know, the storage medium of a mechanical hard disk is a disk, and the head moves on the disk to perform track addressing. The behavior is similar to playing a record. This structure features a high throughput during sequential access. However, if Random Access is made to the disk, a large amount of time will be wasted on moving the head, at this time, the response time of each IO will become longer, greatly reducing the IO response speed. The track-seeking operation of the head on the disk is similar to elevator scheduling. If, during the track-seeking process, the data requests of the relevant track passing by can all be processed by the way, therefore, the overall I/O throughput can be improved with a small impact on the response speed. This is why we asked what I/O scheduling algorithm should be designed. In the beginning, Linux named this algorithm the Linux elevator algorithm. At present, three algorithms are enabled by default in the kernel. In fact, there should be two types of algorithms, because the first one is called noop, which is an empty operation scheduling algorithm, that is, there is no scheduling operation, i/O requests are not sorted, but only a fifo queue for io merge.

Currently, the default Scheduling Algorithm in the kernel is cfq, which is called fully fair queue scheduling. The scheduling algorithm is named like its name. It tries to provide a completely fair I/O operation environment for all processes. It creates a synchronous IO scheduling queue for each process, and allocates IO resources by default in a time slice and a limited number of requests to ensure that the IO resource usage of each process is fair, cfq also implements priority scheduling at the process level, which will be explained in detail later.

You can view and modify the IO scheduling algorithm by using the following methods:

[zorro@zorrozou-pc0 ~]$ cat /sys/block/sda/queue/scheduler noop deadline [cfq] [zorro@zorrozou-pc0 ~]$ echo cfq > /sys/block/sda/queue/scheduler

Cfq is a good IO scheduling algorithm choice for general servers, and is also a good choice for desktop users. However, it is not very suitable for many scenarios with high IO pressure, especially when I/O pressure is concentrated on some processes. In this scenario, we need to meet the IO response speed of one or more processes, rather than making all processes use IO fairly, such as database applications.

Deadline Scheduling (deadline scheduling) is a solution that is more suitable for the above scenarios. Deadline implements four queues, two of which process normal read and write, sort by sector code, and merge normal io to improve throughput. because IO requests may be concentrated in some disk locations, new requests may be merged and io requests from other disk locations may be starved to death. Therefore, two other queues for processing timeout read and write are implemented, which are sorted by the request creation time. If there are timeout requests, they are put into these two queues, the scheduling algorithm ensures that requests in the queue that have timed out (reached the end time) are processed preferentially to prevent requests from starved to death.

Not long ago, four algorithms were used by default in the kernel, and Anticipatory schedory. A tall name made me think that the Linux kernel would be a fortune teller. The results show that it takes only a short time before I/O Scheduling Based on the deadline algorithm. If I/O requests can be merged during this time, they can be merged for processing, improve the data throughput of deadline scheduling in the case of sequential read/write. In fact, this is not a prediction at all. I think it is better to call it a collision scheduling algorithm. Of course, this strategy has a bad effect in some specific scenarios. However, in most scenarios, this scheduling not only does not increase the throughput, but also reduces the response speed. Therefore, the kernel simply deletes it from the default configuration. After all, the purpose of Linux is to be practical, and we will not pay much for this scheduling algorithm.

CFQ fully fair queue

CFQ is the default I/O scheduling queue for the kernel. It is a good choice for desktop application scenarios and most common application scenarios. How to implement a so-called Completely Fair Queueing )? First, we need to understand who the so-called fairness is? From the perspective of the operating system, the subjects that produce operational behaviors are processes. Therefore, the fairness here is for every process. We need to try to make it fair to occupy IO resources. So how can we make the process occupy I/O resources fairly? We need to first understand what I/O resources are. When measuring an I/O resource, we generally like to use two units, one being the data read/write bandwidth, and the other being the data read/write IOPS. Bandwidth is the read/write data volume in the unit of time, for example, 100 Mbyte/s. IOPS is the number of reads and writes per unit of time. In different read/write situations, the performance of the two units may be different, but it is certain that any of the two units has reached the performance limit and will become the bottleneck of IO. Considering the structure of a mechanical hard disk, if read/write is sequential read/write, IO performance can be achieved through a relatively small amount of IOPS, because it can merge a lot of IO, data reading efficiency can also be accelerated through pre-reading and other methods. When I/O performance is biased towards random read/write, the IOPS will become larger, and the merging possibility of I/O requests will decrease. When each IO request has less data, the lower the bandwidth performance. Here we can understand that there are two main forms of IO resources for processes. The number of IO requests submitted by processes per unit time and the bandwidth occupied by processes. In fact, no matter which one is, it is closely related to the IO processing time allocated by the process.

Sometimes the business can occupy a large amount of bandwidth with a small amount of IOPS, while others may occupy less bandwidth with a large amount of IOPS, therefore, it is relatively fair to schedule the IO time used by processes. That is to say, whether you have high IOPS or high bandwidth usage, you can change the process to another process at the time. What do you like. Therefore, cfq tries to allocate a time slice equivalent to that used by Block devices to all processes. In the time slice, a process can submit the generated IO requests to the block device for processing, and the time slice ends, the process requests are arranged in its own queue and will be processed during the next scheduling. This is the basic principle of cfq.

Of course, there cannot be real "fairness" in real life. In common application scenarios, we are willing to manually specify the priority of IO occupation of processes, this is like setting a priority for the CPU usage of a process. Therefore, in addition to fair queue Scheduling for time slices, cfq also provides priority support. Each process can set an I/O priority. cfq serves as an important reference for scheduling based on the priority setting. Priority is first divided into three categories: RT, BE, and IDLE, which are Real Time, Best Try, and Idle, for each category of IO, cfq uses different policies for processing. In addition, the RT and BE categories are divided into eight sub-priorities to achieve more detailed QOS requirements, while the IDLE has only one sub-priority.

In addition, we all know that the core reads and writes to the storage are cached (buffer/cache) by default. In this case, cfq cannot distinguish the process from which the current request is processed. Cfq can distinguish the process from which the IO request comes from only when the process uses synchronous (sync read or sync wirte) or Direct IO (Direct IO) for read/write. Therefore, in addition to the IO queues implemented for each process, a public queue is also implemented to process asynchronous requests.

At present, the kernel has implemented cgroup resource isolation for IO resources. Therefore, cfq also provides scheduling support for cgroup based on the above system. For more information about the blkio function of cgroup, see my previous article on Cgroup-Linux IO resource isolation. In general, cfq uses a series of data structures to achieve the support of all the above complex functions, you can see its implementation through the source code, the file under the source code directory block/cfq-iosched.c.

CFQ Design Principle

Here, we will briefly describe the overall data structure: first, cfq maintains the entire scheduler flow through a data structure called cfq_data. In a cfq that supports the cgroup function, all processes are divided into several contral groups for management. Each cgroup has a cfq_group structure in cfq to describe. All cgroups are put into a red/black tree as a scheduling object and sorted by vdisktime as the key. Vdisktime records the io time occupied by the current cgroup. During each cgroup scheduling, the cgroup with the least time of the Current vdisktime is always selected from the red/black tree for processing, to ensure that the IO resource usage between all cgroups instances is "fair ". Of course, we know that cgroup can allocate resources in proportion to blkio. The principle of cgroup is that cgroup with a large allocation proportion takes a relatively slow increase in vdisktime and vdisktime with a small allocation proportion increases rapidly, the speed is proportional to the allocation ratio. In this way, the IO ratios of different cgroups are different, and the cfq is still "fair.

After selecting the cgroup (cfq_group) to be processed, the scheduler needs to select the next service_tree. The service_tree data structure corresponds to a series of red and black trees, which are mainly used to classify request priorities, such as RT, BE, and IDLE. Each cfq_group maintains seven service_trees, which are defined as follows:

struct cfq_rb_root service_trees[2][3];struct cfq_rb_root service_tree_idle;

Service_tree_idle is the red/black tree used to queue IDLE-type requests. In the preceding two-dimensional array, the first dimension implements an array for RT and BE respectively. Each array maintains three red and black trees, which correspond to three different child types of requests respectively, SYNC, SYNC_NOIDLE, and ASYNC. We can think that SYNC is equivalent to SYNC_IDLE and corresponds to SYNC_NOIDLE. Idling is a mechanism that cfq is designed to merge continuous IO requests as much as possible to increase throughput. It can be understood as a "idling" Wait Mechanism. Idling means that when a queue processes a request, it will take a short time before scheduling. If the next request arrives, the head addressing can be reduced, continue to process the sequential IO requests. To implement this function, cfq implements the SYNC queue in the service_tree layer data structure. If the request is a synchronous ordered request, it will be added to the service tree. If the request is a Synchronous Random request, then, the system enters the SYNC_NOIDLE queue to determine whether the next request is an ordered request. All asynchronous write requests will be sent to the ASYNC service tree, and there is no idling waiting mechanism for this queue. In addition, cfq also has special adjustments to hard disks such as SSD. When cfq finds that the storage device is a queue with a larger depth such as an ssd hard disk, all idling requests for separate queues will not take effect, and all IO requests will join the SYNC_NOIDLE service tree.

Each service tree corresponds to several cfq_queue queues, and each cfq_queue queue corresponds to a process. This will be detailed later.

Cfq_group also maintains an asynchronous IO Request queue shared by all processes in the cgroup. Its structure is as follows:

struct cfq_queue *async_cfqq[2][IOPRIO_BE_NR];struct cfq_queue *async_idle_cfqq;

Asynchronous requests are also divided into three types for processing: RT, BE, and IDLE. Each type corresponds to a cfq_queue for queuing. BE and RT also support priority. Each type has so many IOPRIO_BE_NR priorities. This value is defined as 8 and the array subscript is 0-7. The kernel code we have analyzed is Linux 4.4. From the cfq perspective, we can see that cgroup supports asynchronous IO, we need to define the meaning of the so-called asynchronous IO here. It only means synchronizing data from the buffer/cache in the memory to the hard disk's IO request, instead of aio (man 7 aio) or linux native asynchronous io and libaio mechanisms, in fact these so-called "Asynchronous" IO mechanisms, it is implemented synchronously in the kernel (in essence, the von noiman computer does not have a real "Asynchronous" mechanism ).

As we have already stated above, because the process normally writes data to the buffer/cache first, This asynchronous IO is uniformly processed by the async Request queue in cfq_group. So why should we implement an ASYNC type in the service_tree above? This is of course to support asynchronous IO differentiation of processes and make it "completely fair. In fact, in the blkio System of the latest cgroup v2, the kernel already supports cgroup speed limiting for buffer IO, and the above may be confusing with a bunch of types, all are the type tags needed in the new system. The new system is more complex and more powerful, but don't worry. The official cgroup v2 system will officially meet you when Linux 4.5 is released.

We continue to select service_tree. The service_tree of the three priority types is selected based on the Type priority. The RT priority is the highest, the BE priority is the lowest, and the IDLE is the lowest. That is to say, if there is one in RT, it will continue to process RT, and RT will not process BE. Each service_tree corresponds to a red/black tree whose element is cfq_queue, and each cfq_queue is the request queue created by the kernel for the process (thread. Each cfq_queue maintains an rb_key variable, which is actually the IO service time of the queue ). Here, we still find the cfq_queue with the shortest service time in the red/black tree to serve the service, so as to ensure "completely fair ".

After cfq_queue is selected, I/O requests in the queue will be processed. The scheduling method is similar to deadline. Cfq_queue queues each request that enters the queue twice. One is put into the fifo, and the other is put into the red/black tree that uses the access sector order as the key. By default, requests are retrieved from the red and black trees for processing. When the request delay reaches deadline, the longest waiting time is retrieved from the red and black trees for processing to ensure that requests are not starved to death.

This is the entire cfq scheduling process. Of course, there are still many details, such as merge processing and sequential processing.

CFQ Parameter Adjustment

Understanding the entire scheduling process helps us determine how to adjust the cfq parameters. All cfq adjustable parameters can be found in the/sys/class/block/sda/queue/iosched/directory. Of course, on your system, replace sda with the corresponding disk name. Let's take a look at everything:

[root@zorrozou-pc0 zorro]# echo cfq > /sys/block/sda/queue/scheduler[root@zorrozou-pc0 zorro]# ls /sys/class/block/sda/queue/iosched/back_seek_max  back_seek_penalty  fifo_expire_async  fifo_expire_sync  group_idle  low_latency  quantum  slice_async  slice_async_rq  slice_idle  slice_sync  target_latency

Some of these parameters are related to the track-Finding Method of the mechanical hard drive head. If they indicate that you do not understand them, add the following knowledge:

Back_seek_max: Maximum range for backward addressing of the head. The default value is 16 Mb.

Back_seek_penalty: Penalty Factor of backward addressing. This value is compared with forward addressing.

These two parameters are set to prevent slow addressing caused by head seek jitter. The basic idea is that when an io request arrives, cfq will estimate the head tracing cost based on its addressing position. First, set a maximum value back_seek_max. For a request with a fan ID located behind the head, cfq will process the request as long as the addressing range does not exceed this value. Then, set a coefficient back_seek_penalty to evaluate the cost. When the distance from backward addressing is 1/2 (1/back_seek_penalty) relative to the forward addressing of the head, cfq considers the two request addressing costs to be the same. These two parameters are actually the condition limit for cfq to determine the processing of request merging. All requests that compose this condition will be merged as much as possible during the processing of this request.

Fifo_expire_async: Set the timeout value for asynchronous requests. Synchronous requests and asynchronous requests are processed in different queues. During scheduling, cfq generally processes synchronous requests first and then processes asynchronous requests, unless the asynchronous request meets the restrictions of the preceding merge processing conditions. When the queue of this process is scheduled, cfq will first check whether asynchronous request timeout exists, that is, exceeds the limit of the paio_expire_async parameter. If yes, a time-out request is sent first, and other requests are still processed based on the priority and slice number.

Fifo_expire_sync: this parameter is similar to the preceding one. The difference is that it is used to set the synchronization request timeout time.

Slice_idle: The parameter sets a waiting time. This allows cfq to wait for a while switching cfq_queue or service tree to increase the throughput of the HDD. Generally, IO requests from the same cfq_queue or service tree have better addressing locality, so this can reduce the number of disk addressing times. The default value is non-zero. Of course, setting this value to a non-zero value on a solid state drive or hard RAID device will reduce the storage efficiency. Because SSD does not have the concept of head addressing, it should be set to 0 on such a device, disable this function.

Group_idle: this parameter is similar to the same one. The difference is that when cfq needs to switch to cfq_group, it will wait for a while. In the cgroup scenario, if we use the slice_idle method, the idling wait may occur during the cfq_queue switch of each process in the cgroup group. In this case, if the process has been processing requests, other processes in the same group may not be scheduled until the cgroup quota is exhausted. This will cause other processes in the same group to starve and cause IO performance bottlenecks. In this case, we can set slice_idle = 0 and group_idle = 8. In this way, the idling wait is performed in cgroup rather than cfq_queue processes to prevent the above problems.

Low_latency: This is used to enable or disable the low latency mode of cfq. When this switch is enabled, cfq recalculates the slice time of each process based on the target_latency parameter settings. This will benefit the fairness of throughput (by default, the fairness of time slice allocation ). If this parameter is disabled (set to 0), the value of target_latency is ignored. This will allow the process in the system to allocate IO resources completely in the form of time slice. This switch is enabled by default.

We already know that cfq is designed with the concept of "idling" to allow continuous read/write operations to be merged as much as possible, reducing head addressing operations to increase throughput. If a process always performs sequential read/write quickly, the response speed of other processes that need to process I/O is reduced because of the high hit rate of the empty-to-Wait cfq, if another process to be scheduled does not issue a large number of sequential IO behaviors, the IO throughput of different processes in the system will be unbalanced. For example, when many dirty pages in the cache in the system need to be written back, the desktop needs to open a browser to perform operations. At this time, the background behavior of dirty pages written back may hit a lot of idling time, as a result, the browser waits for a small amount of IO, making the user feel that the browser's response speed is slow. This low_latency is mainly an option for optimization in this case. When it is enabled, the system will restrict the processes that occupy a large amount of IO throughput due to the blank forwarding, to achieve a relatively balanced IO throughput of different processes. This switch is suitable for desktop applications.

Target_latency: when the value of low_latency is enabled, cfq recalculates the length of the IO time slice allocated by each process based on this value.

Quantum: this parameter is used to set the number of IO requests processed from cfq_queue each time. In the event processing cycle of a queue, IO requests exceeding this number will not be processed. This parameter is only valid for Synchronous requests.

Slice_sync: When a cfq_queue queue is scheduled for processing, the total time it can be allocated is specified as a computing parameter using this value. Formula: time_slice = slice_sync + (slice_sync/5 * (4-prio )). This parameter is valid for synchronization requests.

Slice_async: this value is similar to the one that is valid for asynchronous requests.

Slice_async_rq: this parameter is used to limit the maximum number of asynchronous requests that a queue can process within the time range of an slice. The maximum number of requests processed is also related to the io priority set by the relevant process.

Cfq iops Mode

We already know that by default, cfq uses priority-based scheduling supported by time slices to ensure the fairness of IO resource usage. High-priority processes get more time slice lengths, while low-priority process time slice is relatively small. When our storage is a high-speed and NCQ (native command queue) device, we 'd better allow it to process multiple requests from multiple cfq queues, to improve the utilization of NCQ. In this case, it is out of date to use the time slice allocation method to allocate resources, because based on the time slice allocation, there is only one request queue that can be processed at the same time. In this case, we need to switch the cfq mode to the IOPS mode. The switching method is simple, that is, set slice_idle to 0. The kernel automatically checks whether your storage device supports NCQ. If yes, cfq automatically switches to IOPS mode.

In addition, in the default priority-based time slice mode, we can use the ionice command to adjust the IO priority of processes. The IO priority assigned by the process by default is calculated based on the nice value of the process. The calculation method can be seen in man ionice.

DEADLINE Scheduling

The deadline scheduling algorithm is much simpler than the cfq algorithm. The design goal is to ensure that requests are accessed in the order of the device sector, while ensuring that other requests are not starved to death, they must be scheduled before a deadline. We know that the head can perform sequential access and random access to the disk. Because of the relationship between the tracing Delay Time, the IO throughput is higher During sequential access and the random access throughput is smaller. If we want to optimize the throughput for a mechanical hard disk, we can sort the IO requests accessed by the scheduler in a composite order as much as possible, and then send the requests to the hard disk in this order, i/O throughput can be increased. However, there is another problem in this case, that is, if a request occurs at this time, the track to be accessed is far away from the track where the current head is located, and a large number of application requests are concentrated near the current track. As a result, a large number of requests will always be merged and processed by the queue, and the request to access the remote track will be starved to death because it cannot be scheduled. Deadline is such a scheduler that can schedule remote requests within a certain period of time while ensuring the maximum I/O throughput.

DEADLINE Design Principle

To achieve this goal, the deadline scheduler implements two types of queues, one of which is responsible for sorting requests by access sector. This queue uses a red/black tree, called sort_list. The other is to sort the request access time. The linked list is used to organize objects.

Due to the obvious difference in the processing of read/write requests, in each queue, two queues are divided by the read/write type of the request, that is, the deadline scheduler actually has four Queues:

The reason why deadline separates read/write queues is that the read operation has a higher priority than the write operation. From the application perspective, read operations are generally synchronous, that is, during reading, the program generally waits until the data is returned before proceeding to the next step. The synchronization requirements for write operations are not obvious. Generally, programs can write data to the cache, and then the kernel is responsible for synchronizing the data to the storage. Therefore, optimization of read operations can significantly benefit. Of course, in this case, deadline must consider the situation where write operations will starve to death to ensure that they will not starve to death.

Deadline is easy to join: after a new IO request is generated and necessary merging operations are performed, in the deadline scheduler, the scheduler queues sort_list and paio_list respectively based on the slice sequence and request generation time. Then, the data is further queued to the corresponding read/write Queue Based on the read/write type of the request.

Deadline is a little more difficult to team out:

The entire processing logic is like this. The principle is that the read priority is higher than the write priority, and the request processing that reaches the deadline time is higher than the sequential processing. Under normal circumstances, sequential read/write is ensured, throughput is ensured, and hunger is handled in case of hunger.

DEADLINE Parameter Adjustment

Deadline has fewer adjustable parameters, including:

[root@zorrozou-pc0 zorro]# echo deadline > /sys/block/sdb/queue/scheduler[root@zorrozou-pc0 zorro]# ls /sys/block/sdb/queue/iosched/fifo_batch  front_merges  read_expire  write_expire  writes_starved

Read_expire: Set the Read Request timeout value in ms. When a Read Request enters the deadline, its expiration time is set to the current time + read_expire, and is sorted in sort o_list.

Write_expire: Set the Write Request timeout time in ms. Function root read requests are similar.

Restore o_batch: When sort_list requests are processed, deadline will process the requests in batch units. The number of requests processed by each batch is limited by this parameter. During a batch process, no timeout check is generated, and no extra disk seek time is generated. This parameter can be used to balance the conflict between sequential processing and hunger time. When hunger time needs to meet expectations as much as possible, we can reduce this value, this allows you to check as much hunger as possible and handle it in a timely manner. Increasing this value will also increase throughput, but will lead to a longer delay in processing hunger requests.

Writes_starved: this value is used for check during the first step of deadline processing. It is used to determine whether the hunger level of the write queue is high enough when the read queue is not empty. When deadline gives up the processing of the Read Request and processes the write request. When a write request exists, deadline does not process the write request immediately. Instead, it accumulates starved in the relevant data structure. If this is the first time that a write request is checked for processing, then the count is 1. If the value of writes_starved is 2 at this time, we think the hunger level is not high enough at this time, so we can continue to process read requests. Deadline only returns to process write requests when starved> = writes_starved. It can be considered that this value is used to balance the priority State of the read/write request processing by deadline. The larger the value is, the longer the write request is processed, the smaller it is, the higher the Write Request, the higher the Read Request priority.

Front_merges: when a new request enters the queue, if the requested slice is very close to the current slice, it can be merged for processing. However, this merge may be in two cases: merge after the current position and merge before the current position. In some scenarios, the forward merge operation is unnecessary, so we can use this parameter to disable the forward merge operation. By default, deadline supports forward merge and is set to 0.

NOOP Scheduler

Noop scheduler is the simplest scheduler. In essence, it is a fifo queue implemented by a linked list and simple merge processing of requests. The scheduler does not provide any suspicious configuration parameters.

Application scenarios of various schedulers

Based on the analysis of the above several io scheduling algorithms, we should be able to have some general ideas on the Use scenarios of various scheduling algorithms. In principle, cfq is a general scheduling algorithm, which is a Scheduling Algorithm Based on processes to ensure that everyone is as fair as possible. Deadline is a scheduling algorithm based on improving the throughput of a mechanical hard disk. It tries its best to ensure that the scheduling is performed when the input/output requests reach the deadline, it is very suitable for businesses with a single business and heavy I/O pressure, such as databases. What about noop? In fact, if we extend our thinking objects to solid state disks, you will find that both cfq and deadline are queue algorithm adjustments for the structure of the mechanical hard disk, this adjustment is meaningless for SSD. For SSD, the more complicated the IO scheduling algorithm is, the more logic it needs to process and the lower the efficiency. Therefore, the use of noop in SSD is the best, followed by deadline, while cfq is undoubtedly the lowest efficiency due to complexity.

 

 

Reprinted on http://liwei.life/2016/03/14/linux_io_scheduler/