Linux Kernel kprobes Debugging technology __linux

Kernel developers have been trying to find a fast and efficient kernel debugging tool for kernel development.    The efficient debugging technology can improve the kernel development efficiency and shorten the kernel development cycle. This paper studies a new type of kernel debugging technology ―kprobes, Kprobes is a lightweight kernel debugging tool, using Kprobes technology can dynamically insert probe points in the running kernel and perform user predefined operations at the probe point. Based on the source code of Kprobes in Linux kernel, this paper studies the application of Linux CPU anomaly technology, Single-step technology, Loadable Kernel module technology and RCU synchronization technology in Kprobes. Secondly, the implementation of three kinds of debugging methods, such as the Kprobe,jprobe,kretprobe supported by Kprobes, is analyzed and studied in detail.

First, Kprobes Debugging technology

Kprobes Debugging Technology Overview
    in the past, kernel developers have been trying to find a fast and efficient kernel debugging tool for kernel development. Starting with the 2.6 version of Linux, a new kernel debugging technology has emerged, which is kprobes technology. The
    kprobes was first developed by IBM's Dprobe project, and Dprobe is a kernel debugging tool developed by IBM. Starting from the 2.6.9 linux kernel, Kprobes is added to the kernel source code, and is in continuous improvement, more and more functions are added to  Kprobes  kernel debugging technology. kprobes  has been able to work properly on CPU platforms such as I386,X86_64, PPC64 and IA64,SPARC64.

    Currently, most distributions contain a kernel debugging tool SYSTEMTAP. The Systemtap can be scripted to debug the kernel, a tool that relies on  Kprobes  to implement it. kprobes  is a lightweight kernel debugging tool, which means that Kprobes's operations do not affect the flow of normal kernel execution.
    use Kprobes technology to dynamically insert probe points in the running kernel, which can execute user-predefined callback functions when the kernel runs to the probe point. When the user function is finished, it will return to the normal kernel execution flow and start a new round of debugging work.  

    kprobes supports three types of probes: The first is the most basic method of detection, called Kprobe, which supports the placement of probe points at any point in the kernel (in addition to the code associated with the  kprobes implementation). The second type of debugging is called Jprobe, which is used primarily for parameters passed in by the debug function. The third way of debugging is called Kretprobe, which is to execute a user callback function when the function returns, and to debug the kernel function return value. The two debugging methods discussed above are based on the first Kprobe debugging method. Kprobes also supports three types of callback functions: The first is called Pre_handler, which is used to execute before the probe instruction is executed. The second is called Post_handler, which is used to execute after the probe instruction is executed. The third is called Fault_handler, which is used for processing when memory access errors occur.
    currently Kprobes kernel debugging technology has been adopted by many kernel developers and used at various stages of the kernel development process. The development of the
    kprobes is still ongoing, and the current SYSTEMTAP community is responsible for the maintenance of the debugging technology and the development of new functions, mainly by ibm,intel,redhat and other companies to maintain.

Kprobes Configuration Instructions
Before using Kprobes for kernel debugging, you need to configure the kernel to be debugged. Take the Linux 2.6.20.3 version of Linux kernel for example:
First, you need to compile the Kprobes-related code into the kernel, run the Make Menuconfig command in the kernel directory, and select Kprobes in the Instrumentation Support project.
Second, select the Include all symbols in kallsyms entry in Configure standard kernel features, which is used to enable the Kallsyms_lookup_name () function. This function retrieves the address of the kernel function.
In the latest version of the Kprobes implementation, it has been supported to register directly using the function name.
Third, select the Enable Loadable module support entry in Loadable module support, which is used to enable pluggable module functionality for the kernel. Since Kprobes debugging is implemented through module inserts, the debugger needs to select this option by writing a debug module and inserting the kernel for kernel debugging.

The key technology in Kprobes
Kprobes is not only the implementation of pure software, the technology is closely connected with specific hardware, the use of some hardware features. Therefore, the implementation framework of Kprobes is divided into two parts: the first part is the management of Kprobes, which is not architecture-independent code. The second part is the implementation code related to the specific CPU architecture, such as preparing one step execution environment. This code is closely linked to a specific CPU architecture, so there are different implementations on different CPUs. All of the following discussions in this article are based on the Intel IA32 CPU architecture, the 2.6.20.3 kernel. The four key technologies used to support Kprobes implementations are discussed below.

Linux CPU Exception handling technology and its application in Kprobes
CPU exceptions are caused by the fact that the peripheral hardware emits a break signal or performs a CPU exception instruction during CPU operation.
CPU exceptions can be divided into hardware exceptions and software exceptions.
Hardware exceptions are also known as hardware interrupts, which are typically caused by peripheral hardware devices that emit interrupt signals. When a peripheral device emits an interrupt signal, the signal is sent to interrupt processor quorum, the more famous 8259 interrupt processing chip, currently Intel's CPU is generally used APIC (Advanced programmable Interrupt controllers) to achieve. Interrupts the processor to the CPU after it is arbitrated, and the CPU begins an interrupt processing process.
Software anomalies are not emitted by peripheral devices, but are caused by some CPU exception instructions written by programmers, such as INT,INT3, which can cause a software exception. The implementation of Linux system calls on earlier CPUs is the use of int directives. When the CPU executes to these exception directives, it also begins an exception handling process.
The implementation of operating system CPU exception handling is closely related to a particular CPU architecture. Each interrupt or exception to the CPU of the Intel IA32 system has a vector number from 0 to 255. In the Linux kernel, each of the broken vector numbers corresponds to an item in the Interrupt Descriptor Chart (IDT), so there are 256 break descriptors. IDT Each entry contains 8 bytes, and part of these 8 bytes is the address of the interrupt handler function. The meaning of the other fields of the table item is no longer covered here, as can be found in the Intel IA32 Programmer's Manual. The Linux kernel initially interrupts the descriptor table with the Set_trap_gate () macro in the initialization phase.
In the Linux kernel, exception handling is implemented through a level two jump. For example, if you receive an exception signal when the CPU is running, the CPU will find the corresponding item in the Interrupt descriptor table according to the interrupt vector number, and then jump to the address indicated in the table entry to execute.
Here to jump to the address of the general case corresponding to the source file Linux/arch/i386/kernel/entry. An assembly function entry in S. After a certain initialization of the assembly function and then jump to the C function to execute. The C function here is the real exception handler, and when returned from the C function to the original assembly function, the assembly function will do a part of the follow-up work. The final assembly function executes the iret instruction to return to the interrupted code from the interrupt context to continue execution. This is the Linux kernel process exception processing, so that the Linux exception handling process is two times jump implementation.
CPU exceptions are also used in the implementation of Kprobes, and when a probe is inserted, the Kprobes processing routine saves the instructions at the insertion point and replaces it with the int3 instruction. When the CPU runs to the inserted int3 instruction, the Linux kernel enters the exception handling process before running the debugger's predefined callback function. Using CPU anomaly to realize the trigger and processing of detection point is the key of kprobes implementation.

Single-step technology and its application in Kprobes

The individual execution of the debugger is very useful, and the programmer can pinpoint the location of the program error by stepping through the code to determine the process that the program executes, mastering the variable changes at any time. You can say that stepping is a function that a debugger must have. Single-step technology is designed for single step execution of the debugger.
The main idea of Single-step technology is that when a program executes to a single CPU instruction, it generates a CPU exception before execution, at which point the TF (debug bit) position of the CPU's EFlags register at the time the exception is returned is 1, and the if (Interrupt screen bit) Mark position is 0, and then point the EIP to one step execution instruction. When the single-step instruction completes, the CPU automatically produces a debug exception (due to the TF being placed). At this point, the debugger will normally return control to the debugger, back to the interactive mode.
The Single-step technology is also used in the Kprobes implementation, but the goal is not to return to the interactive mode, but to turn control back into the Kprobes control process. When Kprobes completes the Pre_handler () process, the single-step technique is used to execute the debugged instructions. At this point, Kprobes uses the debug exception to execute the Post_handler (). This is single-step.
The main application of technology in Kprobes.

Oadable Kernel module technology and its application in Kprobes

Loadable kernel Module (LKM) technology is another powerful feature of the Linux kernel.
Before the advent of lkm technology, adding kernel code could only be effected by modifying the kernel source and then recompiling the kernel before restarting it.
When lkm technology appears, kernel programming becomes much simpler, and kernel developers just need to write code in the form of a kernel module, and then insert the kernel to run as part of the kernel. Specifically, the LKM technology provides kernel developers with the ability to dynamically insert and unload kernel modules while the kernel is running. LKM technology can extend the functionality of the kernel without recompiling the kernel. At present, a large number of device drivers are implemented using LKM technology. The driver writes the driver in the form of a module and automatically loads it into the kernel when the kernel is started, which can also be implemented by manual loading.
The advent of LKM technology has brought two benefits: first, the LKM technology makes the driver development and debugging become very simple, to a large extent, improve the efficiency of driving development. Second, the advent of lkm technology makes the kernel image not too large, since most of the kernel code can be written as a modular form and loaded only when used. This does not allow the kernel to take up too much space in system memory and affect system performance.
The LKM technology is also used for kernel debugging using Kprobes. The debugger needs to write the debug code as a modular form and insert the kernel. When the debug module is inserted into the kernel, it goes into the debugging phase, and when the Debug module is unloaded from the kernel, it means the end of the debugging process. It can be said that LKM technology is the base of Kprobes technology for kernel debugging.

RCU technology and its application in Kprobes

The Linux kernel sometimes accesses the global data structure, and if the kernel is preempted and the data is modified, or if it is running on an SMP system (that is, a multiprocessor system), there may be multiple CPUs accessing the same memory at the same time, and the kernel data may be in an inconsistency. To avoid this situation, the kernel uses a number of synchronization mechanisms, such as the use of signal synchronization, the use of spin-lock synchronization and other methods. A new kernel synchronization mechanism RCU is introduced in the 2.6 version of the kernel, which is the abbreviation for Read Copy update. The main idea of RCU is divided into two parts, the first part is to prevent other visitors write to the protected object, the second part is to really expand the write behavior. Readers can access objects protected by RCU at any time without obtaining any locks. The writer modifies the copy of the object first, then executes the write behavior when all the readers exit, and the different writers need to sync. The RCU synchronization mechanism is useful for situations where there is a large number of read operations and little write operations. Because in this case, the read operation can read the shared object without obtaining any locks, which greatly improves the efficiency.
There is also a large number of read operations in the operation of the Kprobes to the probe point data structure, and only a very small portion of write operations. In order to improve the efficiency, the RCU mechanism is also introduced in the realization of kprobes. When the detection point registers or logs off, or when the callback function of the probe point is executed, the probe point data structure struct kprobe is protected by the RCU mechanism. According to the RCU principle, the writer's operation is delayed, and if the reader is blocked, the writer's operation is not performed until the reader is awakened, which greatly reduces the efficiency of the RCU. Therefore, the kernel preemption and CPU interrupts are disabled when the Kprobes callback function is executed. For Kprobes, using the RCU mechanism to implement callback function access also greatly improves the efficiency of detecting multiple probe points on SMP systems, because the callback functions can be executed in parallel. But under such a mechanism, a callback function must be designed to be reentrant.

Analysis of Kprobes debugging technology system structure

According to Kprobes source code implementation, from the logic of the basic can be divided into 3 parts: the first part of the registration probe point, the second part of the debugging process, the third part is the cancellation of the detection point part. The first part of the main function is to carry out a number of security checks, and according to the requirements of the installation of detection points. The second part is the key to Kprobes implementation, which performs predetermined operations based on the probe type. There are three main types of detection here: Kprobe,jprobe and Kretprobe. This section also completes actions such as step execution of the probe instruction. The third part is when the debugger cancels the detection request, to the detection point cancellation operation, mainly is restores the detection instruction and so on work.

The following is an introduction to the implementation mechanism of Kprobe,jprobe and Kretprobe respectively.
The realization of 3.1 kprobe
3.1.1 Correlation data structure and function analysis
1) struct KPROBE structure
The data structure is the basis of the entire Kprobes system, and all Kprobes behavior is carried out around the structure. Below are
Main members of the struct KPROBE structure:

2) struct Notifier_block structure
This structure is used to register callback functions that are invoked when an exception occurs. Int (*notifier_call) (struct Notifier_block *, unsigned long, void *) members are callback function pointers, and int priority members are used to set call precedence. The priority defined in Kprobes is the highest priority, ensuring that registered callback functions are called first.

3) Register_kprobe () function
This function is used to complete the registration of the struct KPROBE structure, insert probe points, and so on.

4) Kprobe_handler () function
This function is used to handle INT3 exceptions thrown by Kprobe.

5) Post_kprobe_handler () function
This function is used to handle debug exceptions raised by Kprobe.

Analysis of kprobe processing process

Kprobe is the Kprobes implementation system in the lowest and most basic of a detection mode, Jprobe and Kretprobe detection methods are achieved through the kprobe. The main processing process for Kprobe is shown in the following illustration:

1) Kprobe Registration process
When the debugger inserts a kprobe module into the kernel, the registration probe point operation is performed first. The operation is done primarily by the register_kprobe () function, which is called the Registrar in the following. The parameters of the registrar include a struct KPROBE structure that is created by the debugger in the Debug module.
First, the Registrar will perform some correctness checks to determine whether symbol_name and addr exist in the struct kprobe structure, and if so, return an error. It also determines whether the probe address is in the kernel code snippet and is not in the Kprobes implementation-related code. Such checks are necessary if the probe address appears in the Kprobes implementation-related code, which can result in recursive behavior. If the address being probed has already been registered, it will be organized as a linked list in the kprobe_table.
Second, the Registrar saves the instruction code of the probe address to the ainsn.ainsn of the struct kprobe structure for subsequent single-step operations. When the Kprobe initialization is complete, the registrar inserts the incoming struct KPROBE structure pointer into the hash table. Finally, the registrar replaces the first byte of the detected instruction with the INT3 instruction. Here, Kprobe's registration work is done.

2) Kprobe Int3 exception handling process
Completion of the registration after the completion of the Kprobe, once the kernel is executed to the detected instruction, that is, when the registration is replaced by the INT3 instructions, it will cause a software exception. The CPU executes the interrupt handler function based on the Interrupt descriptor table, and the Int3 interrupt handler function is/linux/arch/i386/kernel/entry. In S, Kprobe_entry (INT3) is the entry point for the interrupt handler function. The assembly interrupt handler function calls the Do_int3 () function as the C-language processing function for INT3 interrupt processing.
The Do_int3 () function begins by calling the Notify_die () function, which is the primary function of calling the exception's callback function that is registered by the kernel code. The Register_die_notifier () is invoked in the initialization code (Init_kprobes () function) of Kprobes to register the exception callback function. The Kprobes registered exception callback function is Probe_exceptions_notify ().
The execution right is now in Kprobes, and the Kprobe_exception_notify () function begins to execute. There is a parameter Val in the parameter of the function that can be used to determine what exception the current callback function is generated from. This exception is generated by the int3 instruction, so the received parameter should be "Die_int3". At this point, the Kprobe_handler () function is called, which is the main implementation function of Kprobes processing INT3 exception. The function first records the address where the exception occurred, because the address is the address of the registered probe point. To prevent the kernel from being preempted, this function prohibits kernel preemption. On the i386 CPU, access to INT3 interrupt processing has shut down the CPU interrupt, the current Kprobes implementation of only i386 system will shut down the CPU interrupt, the implementation of other systems did not do so.
Next, start checking to see if this Int3 exception was raised by the previous Kprobes process, and if it was raised by the previous Kprobes process, there are two possibilities. The first is that the kprobes processing was caused by the detection code executed by the previous callback function, and the second possibility was caused by jprobe, which is discussed in detail in the Jprobe implementation section. If the INT3 exception was not raised by the previous Kprobes process, the registered struct Kprobe structure was found in the hash table based on the previously recorded probe point address. If the structure contains a Pre_handler function pointer, the scheduled function is executed.

When the user-defined Pre_handler function has been executed, part of the debugging work has been completed. Next, start preparing the Single-step step, which is done with the Prepare_singlestep () function. This function is related to architecture, and the following is the main implementation code of the Prepare_singlestep () function on the i386 system CPU:
Program 1 Prepare_singlestep () function part of the code
Regs->eflags |= Tf_mask;
Regs->eflags &= ~if_mask;
Regs->eip = (unsigned long) p->ainsn.insn;
The above code sets the TF bit in the eflags and empties the IF bit and changes the instruction register address returned by the exception to the original Probe command, which takes effect when the exception returns. The Single-step technology has been discussed above, and it is no longer to repeat. After the detected instruction is executed, a CPU exception is thrown at this time due to the CPU's flag register being placed.
Often called debug exceptions in the Linux kernel.

3) The debug exception is handled in the Kprobe debug exception handling
    linux kernel in a similar way to handling the INT3 exception. The interrupt handler function for the DEGUG exception is also in/linux/arch/i386/kernel/entry. In S, Kprobe_entry (Debug) is the entry point of the interrupt handler function for the exception. The function calls the Do_debug () function to further handle the debug exception, and the same Notify_die () function is invoked. Unlike the INT3 exception, the first parameter of the incoming Notify_die () function is "Die_debug".
    Eventually, the Notify_die () function invokes the callback function Kprobe_exceptions_notify () that was registered when the Kprobes was initialized. At this point, control again returned to kprobes . The
    kprobe_exceptions_notify () determines that the incoming type is die_debug, and the Post_kprobe_handler () function is invoked. Post_kprobe_handler () First determines whether the user-defined Post_handler callback function exists and executes if it exists. After
   , the resume_execution () function is called to do some recovery work, which will empty the TF of the Eflages register and do different processing depending on the type of probe instruction. When Resume_execution () is returned, the Post_kprobe_handler () function enables kernel preemption that is prohibited in Int3 exception handling. Here, kprobes the debug exception processing is basically completed, and the control back to the kernel.
    above is the main process of kprobe execution, and you can see that Kprobe has performed user-defined Pre_handler and Post_handler callback functions in a way that uses two CPU exceptions. And the detected instruction is executed by Single-step technology. Once the Kprobe execution cycle completes, it waits for a new round of execution cycle to arrive. Only when the debugger unloads the debug module does the Kprobe life cycle end.

The realization of Jprobe
Correlation data structure and function analysis
1) struct JPROBE structure
This structure is used when registering the Jprobe probe point, which contains two members:
struct Kprobe kp;//This is jprobe an embedded struct kprobe structure member, because Jprobe is implemented based on Kprobe.
kprobe_opcode_t *entry;//This is the proxy function of the probed function.

2) Setjmp_pre_handler () function
The function acts as a pre_handler of the jprobe embedded kprobe, which is invoked first when the probe point is triggered.

3) Longjmp_break_handler () function
The function acts as the break_handler of the Jprobe inline kprobe and is invoked when it enters the INT3 exception again.