Scheduling and synchronization in the kernel

Article Title: Scheduling and synchronization in the kernel. Linux is a technology channel of the IT lab in China. Including desktop applications, Linux system management, kernel research, embedded systems, open source, and other basic classification summaries

This chapter will introduce the various Task Scheduling Mechanisms in the kernel and their logical relationships (here we will cover the concepts of process scheduling, push-back execution, and interruption ), on this basis, we will explain to you the root cause and protection method for Synchronous protection in the kernel. Finally, we provide an example of synchronous access to the shared linked list of the kernel to help you understand the synchronization problems in kernel programming.

Introduction to the relationship between kernel task scheduling and Synchronization

For those who are engaged in application development, the relationship between Task Scheduling and synchronization in the user space is relatively simple, so there is no need to consider the reasons for synchronization too much. This is because each process in the user space has an independent operating space, and the internal data of the process is invisible to the outside. Therefore, even concurrent execution of each process will not compete for data access. The second reason is that the user space is independent from the kernel space, so the user process will not be staggered with the kernel task, so the user process does not have the possibility of concurrent tasks with the kernel. The above two reasons make user synchronization only need to be considered during inter-process communication and multi-thread programming.

However, the situation in the kernel space is much more complicated, and the reason for synchronization needs to be considered is greatly increased. This is because the shared data in the kernel space is visible to all tasks in the kernel, so when accessing data in the kernel, you must consider whether other kernel tasks can be accessed concurrently, whether competition conditions are generated, and whether data synchronization is required. The "culprit" of kernel concurrency is the complex and variable task scheduling in the kernel-the task scheduling here contains all situations that may cause kernel task replacement.

This article does not reiterate the concept of concurrency, competition, and synchronization. The following section describes the general relationship between the above concepts, which is also applicable in the kernel.

Developers of multi-threaded programs often use multiple threads to access shared data and avoid tedious inter-process communication. However, multiple threads may compete for concurrent access to shared data, making the data inconsistent. Therefore, some synchronization methods are required to protect the shared data. Concurrent execution of multiple threads is caused by preemptible scheduling of threads. A thread is interrupted by the scheduling program while accessing shared data, put another thread into operation-if the newly scheduled thread needs to access the shared data, competition will occur. To avoid competition, threads need to access Shared data serially, that is, the access needs to be synchronized-the other party can access the same data only after one party's access to the data ends.

Kernel task

The defined kernel task refers to all the activity objects executed in the kernel. Each kernel task has an independent program counter, stack, and a set of registers. More importantly, they all belong to kernel Scheduling (the scheduling here is generalized and should not be confused with process scheduling) objects, that is, they can be staggered in the kernel.

Kernel task category

Kernel tasks include "kernel Threads", "system calls", "Hardware interruptions", and "half-bottom tasks. Next we will briefly discuss the features of the preceding kernel tasks.

System Call

A system call is a kernel routine executed by a user program through a portal. It runs in the kernel state and is in the process context (the process context includes the process stack and other environments ), it can be considered as a kernel task of a user process, so it has the characteristics of a user-state task, such as executing a process scheduler (schedule ()), can sleep, can access the current process data (through current ). But it is a kernel task, so it cannot be preemptible during execution (before the 2.6 kernel). The system can reschedule other tasks only when the cpu (sleep) is abandoned by itself. (For details about system calls, see system calls)

Hard interrupt task

A hard interrupt is an interrupt that is generated by peripherals other than the processor. The interrupt is received by the processor and sent to the interrupt handler in the kernel for processing. Note: first, the hard interrupt is generated asynchronously and is processed immediately after the interrupt occurs. That is to say, the interrupt operation can seize the code running in the kernel. This is very important. Second, the interrupt operation occurs in the interrupt context (the interrupt context refers to the Context Environment unrelated to any process ). In an interrupted context, you cannot use process-related resources or perform scheduling or sleep. Because scheduling will cause sleep, but sleep must be targeted at the process (sleep is actually marking the Process status, and then pushing the current process into the sleep Queue ), the asynchronous interrupt handler does not know any information about the current process, nor does it care which process is running. It is completely a passthrough. (For more information about hardware interruptions, see hardware interruptions)

Bottom half task

The half-bottom line is entirely due to the influence of the hard interrupt mentioned above. A hardware interrupt task (handler) is an interrupt processing program that quickly, asynchronously, and simply responds quickly to the hardware and completes necessary operations in the shortest possible time. The hard interrupt handler can seize the kernel task and block the same-level or other interruptions during execution. Therefore, the interrupt processing must be fast and cannot be blocked. In this way, it is not suitable to process a task that requires complicated processing. For example, when the network adapter receives data, the network adapter first sends an interrupt signal to the CPU to fetch data, then the system reads data from the network adapter and stores the data in the system buffer, parses the data, and then sends the data to the application layer. If all these operations involve the interrupt handler, the process is obviously too long, resulting in loss of new interruptions. Therefore, Linux developers divide such tasks into two parts: one is called the upper-bottom, that is, the Interrupt Processing Program, which processes hardware-related operations (such as reading data from the NIC to the system cache) in a short and fast manner ); however, if you want to execute tasks with relatively loose time requirements (such as data parsing) in another part, this part is the lower half of what we will talk about here.

The lower half is a type of push-back task, which delays some less urgent tasks to a more convenient time for the system to run. In the kernel, the bottom-half approach has been continuously evolved. At present, we have produced BH, Task queue (Task queues), and Softirq from the original BH (bottom thalf), Tasklet, and Work queue (New in 2.6 kernel ). Next we will introduce their respective features.



Soft Interrupt operation

Soft Interrupt (softirq) is not triggered by hardware interrupt signals as it is, so it is also different from hardware interruption that can be executed at any time. In general, the Soft Interrupt will receive a processing opportunity before returning the user-level program after the kernel processing task is completed. Specifically, it will be executed at three times (do_softirq (): After the hardware interrupt operation is completed, when the system call is returned, and in the kernel scheduler; (In addition, kernel thread ksoftirqd periodic execution Soft Interrupt ). From this we can see that soft interruptions will be followed by hard Interrupt Processing (as if it were fake), so we can seize the kernel task-at least once after the clock is interrupted. Remember that Soft Interrupt can be executed concurrently on different machines.

On a machine with a symmetric multi-processor, two tasks can be executed simultaneously in the critical section. This type is called True concurrency. Comparatively speaking, the concurrency on a single processor does not actually happen at the same time, but is staggered and pseudo-concurrency. However, they both create competitive conditions and require the same protection.

Soft Interrupt is a very underlying mechanism. It is generally used only when the network subsystem and the SCSI subsystem have high performance requirements and require concurrent processing. Although Soft Interrupt is highly flexible and efficient, you must handle complicated synchronization processing (because it can be concurrent on a multi-processor), so it is usually not used directly, it serves as the basis for supporting Tasklet and BH.

It should be noted that the execution of Soft Interrupt is also in the interrupt context, so the interrupt context has the same restrictions as that of hard interrupt.

Tasklet

Tasklet and bottom half are two delay mechanisms built on Soft Interrupt. The difference is that soft interrupt is statically allocated, and similar Soft Interrupt can run concurrently on several CPUs; tasklet can be dynamically allocated, and different types of Tasklets can run concurrently on several CPUs, but similar tasklets cannot; bottom half can only be statically allocated, essentially, the lower half is a high-priority tasklet that cannot be concurrently executed with the other lower half, even if they are of different types and run on different CPUs. Tasklet can be understood as the derivation of Soft Interrupt, so its scheduling time is consistent with that of Soft Interrupt.

Most tasks that require delayed execution in the kernel can be completed using tasklet. Because similar tasklets have already implemented synchronization protection, it is much easier to use tasklet than Soft Interrupt, and the efficiency is also good.

Bottom half

It is the earliest kernel latency method in BH, Which is original, simple, and easy to control, because all BH handlers are strictly executed in sequence-no two BH handlers can be concurrently executed, even if their types are different, in this way, many synchronization protection is reduced during BH execution. But BH had to be eliminated because its "Simplicity" sacrifices the high performance of multi-processor concurrent processing, which is equivalent to the speed at which a team of people crossed the bridge.

Task queue

Task queue is an alternative to BH, and comes from BH, so it has the same attributes as BH. Its original intention is to simplify the operation interface of BH, but its randomness (random quantity, random execution time) has brought chaos to the system, so it has been replaced by the working Queue (in the 2.6 kernel.

However, in the 2.4 kernel, task queues are still widely used, especially scheduling queues, Timer queues, and instant queues (except for the specific task queues that these three systems have taken over, you can also create your own task queue as you like. Of course, you need to schedule it yourself ). Tasks in the scheduling queue are processed during each process scheduling. Tasks in the scheduling queue are processed in the context of the process. The timer queue is processed every time the clock is ticking; an immediate queue can be processed when an interruption is returned or scheduled (so the most efficient processing). They are all processed in the interrupt context.

These task queues are scheduled by a unified kernel thread in the inner core. The thread name is keventd and the process number is 2 (2.4.18 ). You can use the ps command to view the process.

Kernel thread

The kernel thread can be understood as a special process running in the kernel. It has its own "process context" (the context of the user process that calls it), so it is also scheduled by the Process scheduler, you can also sleep-it is similar to the user's process attributes. The difference is that the kernel thread runs in the kernel space and can access the kernel data, which cannot be preemptible during running.

Traditional Unix systems delegate important tasks to periodically executed processes, including refreshing the disk cache, exchanging unused pages, and maintaining network connections. In fact, it is not efficient to execute these tasks in a strictly linear manner. If you place them in the background for scheduling, both functions and end-user processes can get better responses. Because some system processes only run in the Kernel state, the modern operating system delegates their functions to the Kernel Thread, and the Kernel Thread is not dragged down by unnecessary user State context.

Synchronization in the kernel

As long as tasks are staggered in the kernel, The concurrency of shared data is inevitable, and data protection is inevitable. In the final analysis, the reason for job staggered execution in the kernel is still caused by kernel task scheduling. Next we will summarize the reasons for synchronization in the kernel.

Synchronization reason

L interrupt-the interrupt can occur asynchronously at almost any time, and the code being executed may be interrupted at any time.

L sleep and synchronization with user space-processes executed in the kernel may be sleep, which will wake up the scheduler and lead to scheduling of a new user process.

L symmetric multiple processing-two or more processors can execute code simultaneously.

L kernel preemption-because the kernel is preemptible, tasks in the kernel may be preemptible by another task (new capabilities introduced in the 2.6 kernel ).

In the last two cases, the possibility of concurrent execution of kernel tasks is greatly increased, so that concurrency can happen at any time, and the rules are unpredictable.

Concurrency between kernel tasks

In many cases, the preceding kernel tasks can be executed in a staggered manner. Remember, the lower half of a kernel task may actually be executed at any time, so competition is likely to occur (when accessing the same data structure, ). The following describes the possible concurrent operations among these kernel tasks.

It can be abstracted that the general reason for concurrent execution of programs (the same as the kernel state) is nothing more than that of running programs being preemptible by other programs, so we must look at the preemption relationship between kernel tasks:



N interrupt handlers can seize all programs in the kernel (when there is no lock protection), including soft interrupt, tasklet, bottom half, system call, and kernel thread, it even includes hard interrupt handlers. That is to say, the interrupt handler can run concurrently with all these kernel tasks. If the preemptible program and the interrupt handler both need to access the same resource, competition may inevitably occur.

N software interruptions can also seize all the tasks in the kernel. Therefore, data and soft interruptions are shared in kernel code (such as system calls and kernel threads, there will be competition-in addition to hardware interrupt processing programs, there may also be soft interruptions, on the condition that hard interruptions are interrupted by other hard interruptions, the soft interruptions will immediately get the execution opportunity, because the Soft Interrupt is executed after the hard interrupt. In addition, soft interruptions can run concurrently on different processors even for the same type, so sharing data between them will lead to competition. (Soft interruptions cannot be preemptible to each other on the same processor ).

N tasklets of the same type cannot run at the same time, so they do not run concurrently for tasklets of the same type. However, two tasklets of different types may run concurrently on different processors, if there is data sharing between them, competition will occur (tasklet running on the same processor does not compete with each other ).

N Bottom half, regardless of whether it is similar, cannot be concurrently executed on different processors. It is absolutely serialized, so there will never be competition between them. Task queue attributes are basically the same as BH.

N kernel Tasks running in the process context such as system calls and kernel threads may be concurrent with various kernel tasks. Apart from the interrupt (soft and hard) mentioned above, they can be preempted to generate concurrency, it may also voluntarily sleep (for example, in some obstructive operations), discard the processor, and reschedule other tasks, therefore, system calls and kernel threads will compete with other (including their own) system calls and kernel threads in addition to soft and hardware interruptions (half-bottom, etc. Pay special attention to this situation.

Note: tasklet and bottom half are built on Soft Interrupt, so they all follow the Soft Interrupt scheduling rules-they can interrupt the kernel code (System Call) in the process context ), can be interrupted by hard interruptions-in these cases, concurrency may occur.

Kernel synchronization measures

To avoid concurrency and prevent competition. The kernel provides a set of Synchronization Methods to protect shared data. Our focus is not to introduce the detailed usage of these methods, but to emphasize the difference between these methods and them.

The synchronization mechanism used in Linux has been continuously improved since 2.0 to 2.6. From the initial atomic operation to the subsequent semaphores, from the large kernel lock to today's spin lock. The development of these synchronization mechanisms is accompanied by the overprovisioning of Linux from a single processor to a symmetric multi-processor, and the overprovisioning of a non-preemptible kernel to a preemptible kernel. The locking mechanism becomes more effective and more complex.

Currently, sub-operations in the kernel are mostly used for counting. In other cases, the two locks and their variants are most commonly used. One is the spin lock and the other is the semaphore. We will introduce the two lock mechanisms below.

Spin lock

The spin lock is a kind of lock introduced to prevent multi-processor concurrency. It is widely applied to interrupt processing and other parts in the kernel (for single processor, to prevent the concurrency in the interrupt processing, you can simply disable the interrupt without the need to spin the lock ).

A spin lock can only be held by one kernel task at most. If a kernel task attempts to request a spin lock that has been used (held, then this task will always be busy loop-rotate-Wait for the lock to be available again. If the lock is not in contention, the kernel task requesting it will immediately get it and continue. The spin lock can prevent more than one kernel task from entering the critical section at any time. Therefore, this lock can effectively avoid competition for shared resources for Kernel Tasks running concurrently on the multi-processor.

In fact, the intention of the spin lock is to implement lightweight locking in a short period of time. A competing spin lock allows the thread requesting it to spin while waiting for the lock to be available again (especially wasting processing time), so the spin lock should not be held for too long. If you need to lock for a long time, it is best to use a semaphore.

The basic form of the spin lock is as follows:

Spin_lock (& mr_lock );

/* Critical section */

Spin_unlock (& mr_lock );

Because the spin lock can only be held by up to one kernel task at a time point, only one thread is allowed to exist in the critical section at a time point. This satisfies the locking service required by Symmetric Multi-processing machines. On a single processor, the spin lock is only used as a switch to set kernel preemption. If the kernel preemptible does not exist, the spin lock will be completely removed from the kernel during compilation.

Spin locks have many variants in the kernel. For example, for bottom half, you can use spin_lock_bh () to obtain a specific lock and disable half-bottom execution. The opposite operation is executed by the spin_unlock_bh () method. If the access logic in the critical section can be clearly divided into read and write modes, the reader/writer spin lock can be used. The call form is:

Reader code path:

Read_lock (& mr_rwlock );

/* Read-only critical section */

Read_unlock (& mr_rwlock );

Writer code path:

Write_lock (& mr_rwlock );

/* Read/write critical section */

Write_unlock (& mr_rwlock );

To put it simply, spin locks are mainly used in the kernel to prevent concurrent access to the critical zone in the multi-processor and to prevent competition caused by kernel preemption. In addition, the spin lock does not allow the task to sleep (sleep of a task holding the spin lock will cause an automatic deadlock-Because sleep may cause the kernel task holding the lock to be rescheduled, apply for a lock that you already hold), which can be used in the interrupt context.

Deadlock: Assume one or more kernel tasks and one or more resources. Each kernel is waiting for one of these resources, but all resources are occupied. In this case, all kernel tasks are waiting for each other, but they will never release the occupied resources. Therefore, no kernel task can obtain the required resources and continue running, this means that the deadlock has occurred. Self-occupation means that you possess a certain resource, and then apply for the resource that you already possess. Obviously, it is impossible to obtain the resource again.

Semaphores

Semaphores in Linux are sleep locks. If a task tries to obtain an held semaphore, the semaphore will push it into the waiting queue and then sleep it. In this case, the processor is free to execute other code. When the process holding the semaphore releases the semaphore, a task in the waiting queue will be awakened to obtain the semaphore.

The sleep feature of semaphores makes the semaphores suitable for cases where the lock is held for a long time. They can only be used in the process context because the interrupt context cannot be scheduled; in addition, when the Code holds a semaphore, it cannot hold a spin lock.

The basic usage of semaphores is as follows:

Static DECLARE_MUTEX (mr_sem); // declare mutex semaphores

...

If (down_interruptible (& mr_sem ))

/* Interrupted sleep. When the signal arrives, the sleep task is awakened */

/* Critical section... */

Up (& mr_sem );

Like the spin lock, semaphores also have many variants in the kernel, such as the reader-writer semaphores. We will not introduce them here.

Differences between semaphores and spin locks

Although the conditions for use between the two are complex, in fact, in actual use, semaphores and spin locks are not easy to confuse. Note the following principles.

If the code needs to sleep-this often happens when it is synchronized with the user space-using semaphores is the only option. Because it is not restricted by sleep, it is generally easier to use semaphores. If you need to choose between the spin lock and semaphore, it depends on the length of time the lock was held. Ideally, all locks should be held as short as possible, but it is better to use semaphores if they are held for a long time. In addition, unlike the spin lock, semaphores do not disable kernel preemption, so the code holding semaphores can be preemptible. This means that the semaphore will not have a negative impact on the scheduling response time.

Semaphores

???????????????????????????????

Recommended locking method

The spin lock is preferred for low-cost locking.

Short-term locks give priority to spin locks

Semaphores are preferentially used for long-term locking.

Apply spin lock to intercept context locking

Holding locks requires sleep and scheduling to use semaphores.

???????????????????????????????

Introduced from Linux kernel development

There are many ways to prevent concurrency besides the ones mentioned above. We will not detail them here. Having said so much, I hope you will realize that concurrency control is a very difficult issue in kernel programming. to control it, you must clearly understand the scheduling time and characteristics of various tasks in the kernel, in addition, at the initial stage of development, we should be especially careful to protect shared data (all shared data and all data that can be seen by others should be protected). Don't wait until the development is complete to make up for it.

Concurrency Control instance

The following is an example of a multi-core task accessing shared resources. The synchronization methods mentioned above will be used to give you an image of the memory.

The specific scenario of this example is described as follows.

Our main shared resources are linked lists (mine). There are three kernel tasks to operate on it: one is 100 kernel threads (sharelist), which are responsible for transferring new nodes (struct my_struct) from the header) insert a linked list. The second is the timer task (qt_task), which deletes a node from the head of the linked list when every clock is ticking. Third, the system call (the init_exit called by the rmmod command) is responsible for destroying the linked list and uninstalling the module.

We use the module (sharelist. o) to implement the above scenario. When the module is loaded, a timer task will be created in a queue, and the tasks to be executed (task. rounting = qt_task) will be inserted into the timer queue (tq_timer), and then executed repeatedly (but do not stop ). At the same time, the system uses the keventd kernel thread (which is used to execute the task queue, activated by schedule_task, PID = 2) to create 100 kernel threads (create the kernel_thread function) insert the Linked List (completed by sharelist). When the length of the linked list exceeds 100, the node is deleted from the end of the linked list. Finally, when you need to uninstall the module, call the javas_exit function to destroy the entire linked list and complete some operations such as destroying the kernel process we have created.

Next, let's take a look at how to protect our linked list in the program. In the preceding scenario, kernel concurrency includes the concurrency between kernel threads, kernel tasks, and timer tasks. You need to know that the kernel thread is executed in the context of the process, while the timer task is in the lower half, and the execution is in the context of the interruption. Spin locks are required for protection in these two parts of the staggered execution. In our example, the spin_lock_bh () Lock is used to protect the linked list in the execution path of the kernel thread. In the lower part, because the task queue is executed serially and cannot be interrupted by kernel tasks or system calls, therefore, no locks are required. In addition, when you detach a module, the system call and the lower half of the chain table may still be concurrently deleted. Therefore, you need to lock the chain table as described above.

In addition to the use of spin locks for shared linked list access, there are two areas to be synchronized. One is count. This variable belongs to the atomic type and is used to record the id of the linked list contact. The other is to use semaphores to synchronize the creation of threads in the kernel. After the keventd is scheduled, the execution is blocked (down). After the kernel thread is actually started, the execution can continue (up ).

End

Concurrency occurs everywhere, but errors caused by it do not exist every time, because errors caused by the concurrency process usually take one or two steps, therefore, it is "luck" to perform these two steps in a staggered manner, and the chance of errors is sometimes very small. However, the consequences are catastrophic, such as downtime and data integrity destruction. Therefore, we must never take the concurrency lightly. We must take the "take the paper tiger seriously" determination to treat all possible concurrency in kernel code, even programming on a single processor needs to take into account the porting to a multi-processor. In all, be careful.