An understanding of APC

An understanding of APC

Asynchronous Process calling (APCs) is a basic part of the NT asynchronous processing architecture. It is helpful to understand how nt operates and how to execute several core system operations. 1) APCs allows user programs and system components to execute code in the context of a thread in the address space of a process. 2) the I/O manager uses APCs to perform asynchronous I/O operations initiated by a thread. For example, when a device driver calls iocompleterequest to notify the I/O manager that it has ended processing an asynchronous I/O Request, the I/O manager queues an APC to the request initiating thread. Then, at a lower IRQL level, the thread executes APC. APC to copy the I/O operation results and status information from the system space to a buffer of the thread's virtual memory space. 3) You can use APC to obtain or set the context and suspension of a thread. Although APCs is widely used in the NT architecture, there is a lack of documentation on how to use it. This article describes in detail how the NT System Processes APCs and records the exported nt functions so that device-driven developers can use APCs in their programs. I will also demonstrate the implementation of a very reliable nt APC scheduling subroutine kideliverapc to help you better understand the details of APC scheduling. APC object There are two types of APCs in NT: user mode and kernel mode. The user APCs runs in the current context of the target thread in user mode and needs to be run with permission from the target thread. In particular, APCs in user mode requires the target thread to be In the alertable waiting state to be successfully scheduled for execution. By calling any of the following functions, the thread can enter this state. These functions are: kewaitforsingleobject, kewaitformultipleobjects, kewaitformutexobject, and kedelayexecutionthread. In user mode, you can call the sleepex, signalobjectandwait, waitforsingleobjectex, waitformultipleobjectsex, and msgwaitformultipleobjectsex functions to keep the target thread in the alertable waiting state, so that APCS can be executed in user mode, the reason is that these functions eventually call functions such as kewaitforsingleobject, kewaitformultipleobjects, kewaitformutexobject, and kedelayexecutionthread in the kernel. In addition, by calling an undisclosed alert-Test Service ketestalertthread, the user thread can run APCs in user mode. When a user mode APC is shipped to a thread and the preceding wait function is called, if status_user_apc is returned, the kernel switches to control the APC routine when the user mode is returned, after the APC routine is completed, the thread execution continues. Compared with APCs in user mode, APCs in kernel mode runs in kernel mode. It can be divided into two types: general and special. When APCs is delivered to a special thread, APCs in special kernel mode does not need to be licensed from the thread for running. However, the conventional kernel mode APCs requires a specific environment before they are successfully executed. In addition, the special kernel APC is executed as quickly as possible, as long as there are schedulable activities at the apc_level. In many cases, special kernel APC can even wake up blocked threads. The common kernel APC is executed only when all the special APC programs are executed and the target thread is still running, and no other Kernel Mode APC is executed in this thread. User Mode APC is executed only after all kernel mode APC is executed and only when the target thread has the alertable attribute. Each APC waiting for execution exists in a thread execution body, which is managed by the kernel. Each thread in the system contains two APC queues, one for user mode APCs and the other for Kernel Mode APCs. NT describes an APC through a kernel control object that becomes a kapc. Although there is no clear documentation of APCs in the DDK, the APC object is clearly defined in ntddk. h. From the definition of the kapc object below, some do not need to be described. Such as type and size. Type indicates that this is an APC kernel object. In NT, Each kernel object or execution body object has two fields: type and size. The processing function can determine the currently processed object. Size indicates the size of a word-aligned struct. This indicates the memory space occupied by the object. Spare0 seems obscure, but it does not have any profound significance, just to make up the memory. Other domains will be described in the following sections. // Configure //------------------------------------------------------------------------------------------------------- Several function statements and structure definitions: Typedef struct _ kapc { Cshort type; Cshort size; Ulong spare0; Struct _ kthread * thread; List_entry apclistentry; Pkkernel_routine kernelroutine; Pkrundown_routine rundownroutine; Pknormal_routine normalroutine; Pvoid normalcontext; // // N. B. The following two members must be together. // Pvoid systemargument1; Pvoid systemargument2; Cchar apcstateindex; Kprocessor_mode apcmode; Boolean inserted; } Kapc, * pkapc, * restricted_pointer prkapc; //------ APC Environment When a thread executes at any time, assuming that the current IRQL is in the passive level, it may need to temporarily execute code in other process context. To complete this operation, the thread calls the system function keattachprocess. When the system function is returned from this call, the thread runs in the address space of another process. Previously, all APCs waiting for execution in the context of the thread's own process cannot be shipped because the address space of the process to which it belongs is not currently available. However, the new APCs inserted into this thread can be executed in this new process space. Even when the thread is finally separated from the new process, the APCs inserted into the thread can be executed in the context of the process to which the thread belongs. To control APC transmission, each thread in NT maintains two APC environments or states. Each APC environment contains the user-mode APC queue and the kernel-mode APC queue. a pointer to the current process object and three control variables are used to indicate: whether there is a pending Kernel Mode APCS (kernelapcpending), whether there is a general Kernel Mode APC in progress (kernelapcinprogress), and whether there is a pending user mode APC (userapcpending ). these APC environments are stored in the apcstatepointer domain of the thread object. This field is an array composed of two elements. + 0x138 apcstatepointer: [2] ptr32 _ kapc_state Typedef struct _ kapc_state { List_entry apclisthead [maximummode]; Struct _ kprocess * process; Boolean kernelapcinprogress; Boolean kernelapcpending; Boolean userapcpending; } Kapc_state, * pkapc_state, * prkapc_state; Lkd> dt _ kthread Ntdll! _ Kthread + 0x000 header: _ dispatcher_header + 0x010 mutantlisthead: _ list_entry + 0x018 initialstack: ptr32 void + 0x01c stacklimit: ptr32 void + 0x020 Teb: ptr32 void + 0x024 tlsarray: ptr32 void + 0x028 kernelstack: ptr32 void + 0x02c debugactive: uchar + 0x02d state: uchar + 0x02e alerted: [2] uchar + 0x030 iopl: uchar + 0x031 npxstate: uchar + 0x032 saturation: Char + 0x033 priority: Char + 0x034 apcstate: _ kapc_state + 0x04c contextswitches: uint4b + 0x050 idleswapblock: uchar + 0x051 spare0: [3] uchar + 0x054 waitstatus: int4b + 0x058 waitirql: uchar + 0x059 waitmode: Char + 0x05a waitnext: uchar + 0x05b waitreason: uchar + 0x05c waitblocklist: ptr32 _ kwait_block + 0x060 waitlistentry: _ list_entry + 0x060 swaplistentry: _ single_list_entry + 0x068 waittime: uint4b + 0x06c basepriority: Char + 0x06d decrementcount: uchar + 0x06e prioritydecrement: Char + 0x06f quantum: Char + 0x070 waitblock: [4] _ kwait_block + 0x0d0 legodata: ptr32 void + 0x0d4 kernelapcdisable: uint4b + 0x0d8 useraffinity: uint4b + 0x0dc systemaffinityactive: uchar + 0x0dd powerstate: uchar + 0x0de npxirql: uchar + 0x0df initialnode: uchar + 0x0e0 servicetable: ptr32 void + 0x0e4 queue: ptr32 _ kqueue + 0x0e8 apcqueuelock: uint4b + 0x0f0 Timer: _ ktimer + 0x118 queuelistentry: _ list_entry + 0x120 softaffinity: uint4b + 0x124 affinity: uint4b + 0x128 preempted: uchar + 0x129 processreadyqueue: uchar + 0x12a kernelstackresident: uchar + 0x12b nextprocessor: uchar + 0x12c callbackstack: ptr32 void + 0x130 win32thread: ptr32 void + 0x134 trapframe: ptr32 _ ktrap_frame + 0x138 apcstatepointer: [2] ptr32 _ kapc_state + 0x140 previusmode: Char + 0x141 enablestackswap: uchar + 0x142 largestack: uchar + 0x143 resourceindex: uchar + 0x144 kerneltime: uint4b + 0x148 usertime: uint4b + 0x14c savedapcstate: _ kapc_state + 0x164 alertable: uchar + 0x165 apcstateindex: uchar + 0x166 apcqueueable: uchar + 0x167 autoalignment: uchar + 0x168 stackbase: ptr32 void + 0x16c suspendapc: _ kapc + 0x19c suspendsemaphore: _ ksemaphore + 0x1b0 threadlistentry: _ list_entry + 0x1b8 freezecount: Char + 0x1b9 suspendcount: Char + 0x1ba idealprocessor: uchar + 0x1bb disableboost: uchar The main APC environment is located in the apcstate field of the thread object, namely: + 0x034 apcstate: _ kapc_state The APC waiting for execution in the context of the current process in the apcstate queue. At any time, when the nt apc dispatcher and other system components query a thread pending APCs, They will check the main APC environment. If there are any pending APCs, it will be immediately shipped, or its control variable will be modified to be shipped later. The second APC environment is located in the savedapcstate field of the thread object. When the thread is temporarily attached to another process, it is used to back up the master APC environment. When a thread calls keattachprocess and executes subsequent code in another process context, the content of the apcstate domain is copied to the savedapcstate domain. Then, the apcstate domain is cleared, its APC queue is reinitialized, the control variable is set to 0, and the current process domain is set to a new process. These steps are successful to ensure that the APCs waiting in the context space of the process to which the thread belongs are not transmitted and executed when the thread runs in different process contexts. Subsequently, the content of the apcstatepointer domain array is updated to reflect the new state. The first element in the array points to the savedapcstate domain, and the second element points to the apcstate domain, indicates that the APC environment of the process context of the thread is located in the savedapcstate domain. The APC environment of the new process context of the thread is located in the apcstate domain. Finally, the current process context is switched to the new process context. For an APC object, the current APC environment is determined by the apcstateindex domain. The value of the apcstateindex field is used as the index of the apcstatepointer domain array to obtain the target APC environment pointer. Then, the target APC environment pointer is used to store the APC object in the corresponding queue. When a thread is detached from a new process (kedetachprocess), any pending kernel APCs waiting for execution in the new process address space is distributed for execution. Then the content of the savedapcstate domain is copied back to the apcstate domain. The content of the savedapcstate domain is cleared, the apcstateindex domain of the thread is set to originalapcenvironment, And the apcstatepointer domain is updated. The context of the current process is switched to the process to which the thread belongs. Use APCs The device driver uses two main functions to use APCs. The first is keinitializeapc, which is used to initialize the APC object. This function accepts an APC object allocated by the driver, a target thread object pointer, an APC Environment Index (indicating the APC environment where the APC object is stored), the APC kernel, rundown, and normal routine pointer, APC type (user mode or kernel mode) and a context parameter. The function declaration is as follows: Ntkernelapi Void Keinitializeapc ( In prkapc APC, In pkthread thread, In kapc_environment environment, In pkkernel_routine kernelroutine, In pkrundown_routine rundownroutine optional, In pknormal_routine normalroutine optional, In kprocessor_mode apcmode, In pvoid normalcontext ); Typedef Enum _ kapc_environment { Originalapcenvironment, Attachedapcenvironment, Currentapcenvironment } Kapc_environment; Keinitializeapc first sets the type and size fields of the APC object to an appropriate value, and then checks the value of the environment parameter. If it is currentapcenvironment, The apcstateindex field is set to the apcstateindex field of the target thread. Otherwise, the apcstateindex field is set to the value of the environment parameter. Then, the function directly uses parameters to set the values of the APC object thread, rundownroutine, and kernelroutine fields. To correctly determine the APC type, keinitializeapc checks the value of the normalroutine parameter. If it is null, the value of the apcmode field is set to kernelmode, And the normalcontext field is set to null. If the value of normalroutine is not null, it must point to a valid routine at this time, and corresponding parameters are used to set the apcmode and normalcontext fields. Finally, keinitializeapc sets the inserted domain to false. However, the initialization of the APC object does not store it in the corresponding APC queue. From this explanation, you can understand that if the APCs object lacks a valid normalroutine, it will be treated as the kernel mode APCs. In particular, they will be considered as the special Kernel Mode APCs. In fact, I/O manager uses this type of APC to complete asynchronous I/O operations. On the contrary, the APC object defines a valid normalroutine, And the apcmode domain is kernelmode, which will be treated as the conventional kernel mode APCS; otherwise, it will be treated as the user mode APCs. ntddk. in H, kernelroutine, rundownroutine, and normalroutine are defined as follows: Typedef Void (* Pkkernel_routine )( In struct _ kapc * APC, In out pknormal_routine * normalroutine, In out pvoid * normalcontext, In out pvoid * systemargument1, In out pvoid * systemargument2 ); Typedef Void (* Pkrundown_routine )( In struct _ kapc * APC ); Typedef Void (* Pknormal_routine )( In pvoid normalcontext, In pvoid systemargument1, In pvoid systemargument2 ); //------------------ Generally, no matter what type, each APC object must contain a valid kernelroutine function pointer. When this APC is transferred and executed by the nt apc dispatcher, this routine is first executed. APCs in user mode must contain a valid normalroutine function pointer, which must be in the user memory area. Similarly, the regular Kernel Mode APCs must also contain a valid normalroutine, but it runs in the kernel mode just like kernelroutine. As an optional, any type of APC can define a valid rundownroutine. This routine must be in the kernel memory area, and only when the system needs to release the APC queue content,. For example, when a thread exits, in this case, kernelroutine and normalroutine are not executed, and only rundownroutine is executed. APC objects without this routine will be deleted. Remember, the action of shipping APCs to a thread is only completed by the operating system calling kideliverapc. Executing APC is actually calling the routine in APC. Once the APC object is initialized, the device driver calls keinsertqueueapc to store the APC object in the corresponding APC queue of the target thread. This function accepts an APC Object Pointer, two system parameters, and a priority increment that is initialized by keinitializeapc. Like the context parameter passed to the keinitializeapc function, these two system parameters are simply passed to the APC routine when the APC routine is executed. Ntkernelapi Boolean Keinsertqueueapc ( In prkapc APC, In pvoid systemargument1, In pvoid systemargument2, In kpriority Increment ); //----------------- Before keinsertqueueapc stores the APC object in the APC queue of the target thread, it first checks whether the target thread is APC queueable. If not, the function immediately returns false. If yes, the function directly sets the systemargument1 and systemargument2 fields with parameters. Then, the function calls kiinsertqueueapc to store the APC object to the corresponding APC queue. Kiinsertqueueapc only accepts one APC object and one priority increment. This function first gets the spinlock of the APC queue of the thread and holds it to prevent other threads from modifying the APC structure of the current thread. Then, check the inserted field of the APC object. True indicates that the APC object has been stored in the APC queue. The function returns false immediately. if the inserted field of the APC object is false. the function uses the apcstateindex domain to determine the target APC environment, and then stores the APC object in the corresponding APC queue, linking the apclistentry domain of the APC object to the apclisthead domain of the APC environment. The position of the link is determined by the APC type. In general Kernel Mode APC, user mode APC is stored at the end of the corresponding APC queue. Conversely, if some APC objects are already stored in the queue, the special Kernel Mode APC is stored before the first common Kernel Mode APC object in the queue. If the user APC is defined by the kernel when the thread exits, it will also be placed before the corresponding queue. Then, set the userapcpending domain cup in the main APC environment of the thread to true. In this case, kiinsertqueueapc sets the inserted field of the APC object to true, indicating that the APC object has been stored in the APC queue. Next, check whether the APC object is queued to the context APC environment of the thread. If not, the function returns true immediately. If this is a kernel mode APC, set the kernelapcpending field in the main APC environment to true. In the Win32 SDK documentation, APCs is described as follows: When an APC is successfully stored in its queue, a soft interrupt is triggered, APC will be executed in the next time slice where the thread is scheduled to run. However, this is not completely correct. Such a soft interrupt occurs only when a kernel-mode APC (conventional kernel mode APC or special Kernel Mode APC) is targeted at the calling thread. Then the function returns true. 1) if APC is not for the calling thread, the target thread remains in the waiting state at the passive permission level; 2) This is a conventional kernel mode APC 3) This thread is no longer in the critical section. 4) No other general Kernel Mode APC is still in progress The thread is awakened and the returned status is status_kernel_apc. But the waiting status is not aborted. If this is a user mode APC, kiinsertqueueapc checks whether the target thread is in the alertable waiting state and the waitmode field is set to usermode. If yes, set the userapcpending domain of the master APC environment to true. Wait for the status to return status_user_apc. Finally, the function releases the spinlock and returns true, indicating that the APC object has been successfully put into the queue. Early as a supplement to APC management functions, device driver developers can use the undisclosed system service ntqueueapcthread to directly deliver a user-mode APC to a thread. This function actually calls keinitializeapc and keinsertqueueapc to complete this task. Ntsysapi Ntstatus Ntapi Ntqueueapcthread ( In handle thread, In pknormal_routine normalroutine, In pvoid normalcontext, In pvoid systemargument1, In pvoid systemargument2 ); APC dispatcher of NT NT checks whether there are pending APCs in the thread. Then the APC dispatcher subprogram kideliverapc is executed in the context of this thread to start executing the pending APC. Note: This action interrupts the normal execution process of the thread. First, the control is sent to the APC dispatcher. Then, after the kideliverapc operation is completed, the thread continues to be executed. For example, when a thread is scheduled to run, in the last step, the context switching function swapcontext is used to check whether the new thread has a pending kernel APCs. if yes, swapcontext either (1) requests an APC-level Soft Interrupt to start APC execution, because the new thread runs at the low IRQL (passive level. Or (2) return true, indicating that the new thread has a pending kernel APCs. Whether the execution is (1) or (2) depends on the IRQL level of the new thread. if the permission level is higher than the passive level, swapcontext executes (1). If it is in the passive level, select execute (2 ). The returned value of swapcontext is only available to specific system functions. These system functions call swapcontext to forcibly switch the thread context to another thread. then, when these system functions continue after a period of time, they usually check the return value of swapcontext. If it is true, they will call the APC dispatcher to deliver the kernel APCs to the current thread. for example: The system function kiswapthread is waiting for the service to discard the processor until the wait ends. This function calls swapcontext internally. When the wait is over and the execution continues from the swapcontext call, the returned value of swapcontext is checked. If it is true, kiswapthread will lower the IRQL level to the APC level, and then call kideliverapc to execute the kernel APCs in the current thread. For APCs, the kernel calls the APC dispatcher only when the thread returns to the user mode and the userapcpending domain of the main APC environment of the thread is true. For example, when the system service dispatcher kisystemservice completes a system service request and is planning to return to user mode, it checks whether there are pending user APCs. In execution, kideliverapc calls the kernelroutine of the user APC. Then, the kiinitializeuserapc function is called to set the trap frame of the thread. Therefore, when the thread exits in kernel mode, it starts to run in user mode. . The function of kiinitializeuserapc is used to copy the previous execution status of the current thread (when it enters kernel mode, this status is saved in the trap frame created by the thread Kernel stack ), from the kernel stack to the thread user mode stack, initialize the user mode APC. The APC dispatcher subroutine kiuserapcdispatcher is in Ntdll. dll. Finally, load the EIP register of the trap frame and the kiuserapcdispatcher address in Ntdll. dll. When the trap frame is finally released, the kernel transfers the control to kiuserapcdispatcher. This function calls the normalroutine routine of APC, And the normalroutine function address and parameters are in the stack. When the routine is completed, it calls ntcontinue to let the thread continue to execute using the context before the stack, as if nothing had happened. When the kernel calls kideliverapc to execute a user mode APC, the previusmode field in the thread is set to usermode. trapframe domain pointing to the thread's trap frame. When the kernel calls kideliverapc to execute the kernel APCs, The previusmode field in the thread is set to kernelmode. trapframe field pointing to null. Note that whenever kernelroutine is called, the pointer passed to it is a copy of a local APC attribute, because the APC object is out of the queue, therefore, the APC memory can be safely released in kernelroutine. In addition, this routine has a final opportunity to modify these parameters before its parameters are passed to other routines. Conclusion: APC provides a very useful mechanism to allow asynchronous code execution in a specific thread context. As a device driver developer, you can rely on APCs to execute a routine in a specific thread context without the thread's permission and interference. For user applications, the user mode APCs can be used to effectively implement some callback notification mechanisms.