Linux kernel Debugging Method __linux

Linux kernel debugging methods

KDB: Only in the assembly code level for debugging;

The advantage is that two machines are not required for debugging.

GDB: When debugging a module, it lacks some vital features that can be used to view the operation of the kernel, including disassembly kernel functions.

Kgdb: can be very convenient at the source level to debug the kernel, the disadvantage is that KGDB can only remote debugging, it requires a serial line and two machines to debug the kernel (or it can be on the same host with VMware software running two operating systems to debug)

PRINTK () is one of the most common techniques for debugging kernel code. adding PRINTK () debug calls to specific locations in the kernel code allows you to directly print the information you care about to the screen so that you can watch the program's execution path and the variables, pointers, and other information you care about. The Linux kernel debugger (Linux kernel debugger,kdb) is a patch of the Linux kernel that provides a way to check kernel memory and data structures while the system is running. Oops, KDB in the article Master Linux Debugging technology has detailed introduction, we can refer to. Kprobes provides an interface to force access to any kernel routines and to gather information from interrupt handlers without interference. Using Kprobes, you can easily gather debugging information such as processor registers and global data structures, without the need to compile and start frequently on the Linux kernel, refer to using the Kprobes debugging kernel.

/proc File system

In the/proc file system, the read-write operation of a virtual file is a means of communicating with the kernel, and to see the messages in the kernel loop buffer, you can use the DMESG tool (or use the CAT/PROC/KMSG command via the/proc itself). Listing 6 shows the last few messages displayed by DMESG.

Listing 6. View kernel output from LKM

[root@plato]# DMESG | Tail-5
Cs:io Port probe 0xa00-0xaff:clean.
Eth0:link is down
Eth0:link are up, running at 100Mbit Half-duplex
My_module_init called.  The Module is now loaded.
My_module_cleanup called.  The Module is now unloaded.

You can see the message for this module in the kernel output. Now let's take a moment away from this simple example and look at a few kernel APIs that can be used to develop useful LKM.

Debugging Tools

Use the debugger to track your code step-by-step, viewing the values of variables and computer registers. Using an interactive debugger in the kernel is a complex issue. The kernel runs in its own address space. The common features provided by the debugger in many user spaces are difficult to use in the kernel, such as breakpoints and Single-step debugging.

Directory [hide] 1 kernel bug trace 1.1oops message Analysis 1.2 system crash restart 1.2.1 (1) Tool kexec Introduction 1.2.2 (2) kdump Introduction 1.3SysRq Magic Key printing kernel Information 1.4 command strace 1.5 Use function PRINTK Print kernel information 1.6 kernel kprobe 1.7Systemtap debug 1.7.1 (1) Systemtap principle 1.7.2 (2) STAP program 1.7.3 (3) Systemtap script syntax 2kdb kernel debugger 2.1 Installation KDB 2.2 use KDB Debug command 3kgdb 3.1kgdb Debugging principle 3.2 Establish kdbg online debugging method 3.3 Debug Kernel module 3.4 Debug Kernel 4 use UML to debug Linux kernel 4.1UML principle 4.2 Compiling UML mode client linux kernel 4.3 running UML 4.4 Establishing Serial line and console 4.5 establishing network 4.6 sharing file system between virtual machines 4.7 creating UML File System 4.8 Host file access 4.9 kernel debugging 5 Assertion Statement 6 synchronous lock debugging

Kernel bug Tracking Oops Message Analysis

(1) Oops message generation mechanism

Oops (also known as panic), said the program runs a crash, the program crashes will produce oops messages. The crash of an application or kernel thread produces oops messages, which normally occur when the system does not panic and print oops information in a terminal or log.

When a null pointer or incorrect pointer value is used, a oops message is typically raised because the page mapping mechanism fails to send a virtual address image to the physical address when an illegal pointer is referenced, and the processor sends a "page invalidation" signal to the operating system. The kernel cannot "change pages" to an address that does not exist, and the system produces a "oops".

Oops shows the state of the processor when an error occurs, including the contents of the CPU registers, the location of the page descriptor chart, and some of its difficult to understand information. These messages are generated by the PRINTK statement in the failure-handling function (ARCH/*/KERNEL/TRAPS.C). The more important information is the instruction pointer (EIP), the address of the error instruction.

Because it is difficult to see meaning from hexadecimal values, you can use the symbol resolution tool KLOGD. The KLOGD daemon can decode the oops message before it arrives at the record file. KLOGD is run by default and is decoded by symbols.

Typically oops text is read by KLOGD from the kernel buffer and passed to SYSLOGD, which is written to the Syslog file by syslogd, typically/var/log/messages (dependent on/etc/syslog.conf). If the klogd crashes, the user can read the data from the kernel buffer dmesg > file and save it. You can also use "cat/proc/kmsg > file" to read data, at which point the user is required to abort the transmission because kmsg is a "never ending file."

When protection errors occur, the KLOGD daemon automatically translates the important addresses in the kernel log information into their corresponding symbols. KLOGD performs static address translation and dynamic address translation. Static address translation uses the System.map file to translate symbolic addresses into symbols. The KLOGD daemon must be able to find the System.map file when it initializes.

Dynamic address translation usually translates the symbols in the kernel module. The kernel module's memory is allocated from the kernel dynamic memory pool, and the position of the symbol in the kernel module is finalized after the kernel is loaded.

The Linux kernel provides calls that allow programs to decide which modules to load and where they are in memory. Through these system calls, the KLOGD daemon generates a symbol table for debugging protection errors that occur in loadable modules. Kernel module loading or unloading will automatically send signals to KLOGD, KLOGD can translate the address of kernel module symbols dynamically into symbolic strings.

(2) sample code for generating oops

Using null pointers and buffer overflows is the two most common cause of oops. The following two functions, Faulty_write and Faulty_read, are write and read functions in a kernel module that illustrate both cases. When the kernel calls these two functions, a oops message is generated. function Faulty_write Deletes a reference to a null pointer, and since 0 is not a valid pointer value, the kernel prints oops information and then kills the process that calls some functions.

ssize_t faulty_write (struct file *filp, const char _ _user *buf, size_t count, loff_t *pos)
{/
    * make a simple FA Ult by dereferencing a NULL pointer
    * * (int *) 0 = 0;
    return 0;
}

The function Faulty_write produces oops information listed below (note the symbols that have been decoded in the EIP and Stack Trace Records):

Unable to handle kernel NULL pointer dereference in virtual address \

   00000000 

Printing eip:c48370c3 *pde = 00000000 oops:0002 cpu:0 eip:0010:[faulty:faulty_write+3/576] eflags:00010286 eax:ffffffea ebx:c2c55ae0 ecx:c48370c0 edx:c2c55b00 esi:0804d0 edi:0804d038 ebp:c2337f8c esp:c2337f8c ds:0018 es:0018 ss:0018 processcat (pid:23413,stackpage=c2337000) Stack: 00000001 c01356e6 c2c55ae0 0804d038 00000001 c2c55b00 \

          00000001 
     0804d038 Bffffbd4 00000000 00000000 bffffbd4 c010b860 00000001 \ 
          0804d038 
     00000001 00000001 0804d038 bffffbd4 00000004 0000002b 0000002b \ 
          00000004 

Call Trace: [SYS_WRITE+214/256][SYSTEM_CALL+52/56]

CODE:C7 C0 EC 5d C3 8d B6 00 (00)

In the above oops message, string 3/576 indicates that the processor is on the 3rd byte of the function, and the function overall length is 576 bytes. function Faulty_read Copy a string to a local variable, because the string is overrun by the number leader of the destination. When the function returns, the buffer overflow causes the Oops information to be generated. This error is difficult to detect and track because the return instruction causes the instruction pointer to not find the running address.

ssize_t faulty_read (struct file *filp, char _ _user *buf, size_t count, loff_t *pos)
{
    int ret;
    Char stack_buf[4];
    * Let ' s try a buffer overflow
    /memset (Stack_buf, 0xff);
    if (Count > 4)
        count = 4;
    /* Copy 4 bytes to the user *
    /ret = Copy_to_user (buf, Stack_buf, count);
    if (!ret) return
        count;
    return ret;
}

The function Faulty_read produces oops information listed below:

Eip:0010:[<00000000>]

Unable to handle kernel paging request on virtual address ffffffff printing eip:ffffffff oops:0000[#5] SMP cpu:0 eip:0 060:[] Not tainted eflags:00010296 (2.6.6) was at 0xFFFFFFFF eax:0000000c ebx:ffffffff ecx:00000000-edx:bfffda7c ES i:cf434f00 edi:ffffffff ebp:00002000 esp:c27fff78 ds:007b es:007b ss:0068 processhead (pid:2331,threadinfo=c27fe00 0 task=c3226150 stack:ffffffff bfffda70 00002000 cf434f20 00000001 00000286 cf434f00 fffffff7 bfffda70 c27fe000 c0150612 cf434f00 bfffda70 00002000 Cf434f20 00000000 00000003 00002000 c0103f8f 00000003 bfffda70 00002000 00002000 bfffda70 Call Trace:[] sys_read+0x42/0x70[] Syscall_call+0x7/0xb

Code:bad EIP value.

In the above oops message, due to a buffer overflow, only part of the function call stack can be seen, vfs_read and faulty_read are not visible, and the code only outputs "bad EIP value.", the Address "ffffffff" that is listed at the beginning of the stack. Indicates that the kernel stack has crashed.

(3) Oops information analysis

In the face of the resulting oops information, you should first find the location of the source program oops, by looking at the value of the EIP of instruction instruction register, you can find the location, such as: eip:0010:[faulty:faulty_write+3/576].

More information can be found by looking up the function call stack (called stack). Local variables, global variables, and function parameters can be distinguished from the function call stack. For example: In the function faulty_read oops information function call stack, the stack top is ffffffff, the stack top value should be a value less than FFFFFFFF, for this value, the description can not find back the call function address, it is possible because of buffer overflow and other reasons caused pointer error.

On the x86 architecture, the user-space stack starts with the 0xc0000000, and the recursive value bfffda70 may be the stack address of the user's space. It is actually the buffer address passed to the read system call, and the system calls read into the kernel to copy the user space buffer data to the kernel space buffer.

If the Oops information displays the address of the triggering oops as 0XA5A5A5A5, it is most likely due to the absence of dynamic memory initialization.

Also, if you want to see the symbols of the function call stack, turn on the config_kallsyms option when compiling the kernel.

KLOGD provides a lot of information to help with the analysis. In order for KLOGD to work correctly, symbol table file System.map must be provided in/boot. If the symbol table does not match the current kernel, KLOGD rejects the parsing symbol.

Sometimes a kernel error hangs the system completely. For example, the code goes into a dead loop and the system does not respond to any action. At this point, you can prevent a dead loop by inserting a schedule call at some key points. system Crash reboot

Because the kernel is running incorrectly, in some extreme cases, the kernel will crash, and the kernel will panic when it crashes. To solve this problem, the kernel introduces a new kernel mechanism for fast loading and restarting. The kernel triggers a new kernel by Kdump in the crash, storing the old memory image for easy debugging, allowing the system to run on the new kernel, thus avoiding the panic and enhancing the stability of the system. (1) Tool Kexec Introduction

Kexec is a set of system calls that allow the user to load another kernel from the currently executing kernel. The user can use the shell command "Yum install Kexec-tools" To install the Kexec Toolkit, and after installation, the Kexec command is available.

The tool kexec directly into a new kernel, which enables users to mount from the current kernel and launch into another kernel through system calls. In the current kernel, Kexec performs the function of bootloader. The main difference between a standard system boot and a kexec boot is that the firmware or BIOS that relies on the hardware architecture will not be executed for hardware initialization during kexec startup. This will greatly reduce the time to reboot.

In order for the kernel's kexec function to work, the kernel compilation configuration should confirm that the "Config_kexec=y" is selected, and this entry should be visible in the. config file that is generated after the configuration.

The use of tool kexec is divided into two steps, first, to load the debugged kernel into memory with kexec, and then start the loaded kernel with kexec.

The syntax for loading the kernel is listed below:

Kexec-l kernel-image--append=command-line-options--initrd=initrd-image

In the above command, the parameter is kernel-image the kernel-mounted mapping file, which does not support compressed kernel image file Bzimage, and should use uncompressed kernel-mapped file vmlinux; parameter initrd-image use INITRD mapping file for startup The parameter command-line-options is a command-line option, the command-line option from the current kernel, which can be extracted from the file "/proc/cmdline", and the contents of the file are listed below:

^-^ $ cat/proc/cmdline

Ro root=/dev/volgroup00/logvol00 rhgb quiet

For example, the kernel that the user wants to start is mapped to/BOOT/VMLINUX,INITRD as/BOOT/INITRD, then the kexec Load command is listed as follows:

Kexec–l/boot/vmlinux–append=/dev/volgroup00/logvol00 INITRD=/BOOT/INITRD

You can also add the option-P or--load-panic, which means that loading the new kernel crashes in the system kernel.

After the kernel mounts, start the loaded kernel with the following command, and run it in a new kernel:

Kexec-e

When Kexec migrates the current kernel to the new kernel, kexec copies the new kernel to the reserved memory block, as shown in Figure 1, where the original system kernel reserves a piece of memory (in the shadow portion of the figure) to the kexec loading kernel to load the new kernel, while the other memory areas are not loaded with the new kernel. Used by the original system kernel.

Figure 1 The kexec of the loaded kernel in the pre-reserved location diagram

On a x86-architected machine, the system starts with the first 640KB physical memory for kernel mount, and kexec backs up the zone before restarting into the dump capture kernel. Similarly, PPC64-frame machines need to use the first 32KB physical kernel in the boot and need to support 64K pages, kexec back up the first 64KB memory. (2) kdump Introduction

Kdump is a kexec crash dump mechanism (kexec-based Crash dumping), regardless of whether the kernel kernel requires a dump, such as when the system crashes, Kdump uses kexec to quickly launch into the dump capture kernel. Here, the original running kernel is called the system kernel or the original kernel, and the newly mounted kernel is called the dump capture kernel or mount kernel or new kernel.

During the reboot, the memory image of the original kernel is saved, and the captured kernel (the newly mounted kernel) can access the dump's image. Users can use the command CP and SCP to copy a memory map to a dump file on a local hard disk or copy it over a network to a remote computer.

Currently only x86, x86_64, PPC64 and IA64 architectures support Kdump and kexec.

When the system kernel is started, it retains a small portion of the memory to the dump capture kernel, ensuring that direct memory access from the system kernel is in progress (Memory ACCESS:DMA) does not destroy the dump capture kernel. Command kexec–p load the new kernel into this reserved memory.

Before the crash, the core image of all system cores was encoded in elf format and stored in the kernel's reserved area. The starting physical address of the ELF head passes through the parameter elfcorehdr=boot to the kernel of the dump capture.

By using the dump capture kernel, the user can access the memory image or old memory in two ways:

(1) through the/DEV/OLDMEM device interface, the Capture tool program can read the device file and write out the memory in the original stream format, which is a dump of the memory raw stream. The analysis and capture tools must be smart enough to determine where to find the correct information.

(2) through/proc/vmcore, can output dump information in elf format files, users can use the GDB (GNU Debugger) and crash debugging tools and other analysis tools to debug dump files.

(3) Establish quick reboot mechanism and installation tool

1) Installation Tool Kexec-tools

You can download the source code compilation installation Tool Kexec-tools. Because tool Kexec-tools also relies on some other libraries, the best approach is to use the command "Yum install Kexec-tools" To download the installation from the Web and automatically resolve dependencies.

2) Compile system and dump capture kernel

A separate dump capture kernel can be compiled to capture the kernel's dumps. You can also use the original system kernel as a dump to capture the kernel, in which case there is no need to compile a separate dump capture kernel, but only the architecture that supports the relocation kernel can be used as a dump capture kernel, such as: Architecture i386 and IA64 support relocation kernel.

For the system and dump capture kernel, in order to turn on kdump support, the kernel needs to set some special configuration options, below the system kernel and the dump capture kernel configuration options are described:

The configuration options for the system kernel are described below: in the menu entry "Processor type and features." The option "Kexec system call" is turned on to enable the kernel to compile and install Kexe system calls. The configuration file. CONFIG generates the statement "Config_kexec=y". In the menu entry "filesystem"-> "Pseudo filesystems." The option "Sysfs File system support" is turned on to enable the kernel to compile and install the filesystem sysfs. configuration file. config generates the statement "Config_sysfs=y". In the menu entry "Kernel hacking." Opens the option "Compile the kernel with debug info" so that the kernel compiles and installs to support debugging information output, generating debug symbols for analyzing dump files. The configuration file. CONFIG generates the statement "Config_debug_info=y".

The Dump Capture kernel configuration option (not dependent on the processor architecture) is described as follows: Open the option "kernel crash dumps" in the menu entry "Processor type and Features", configuration file. config statement "Config_crash_ Dump=y ". Open the option "/proc/vmcore support" in the menu entry "filesystems"-> "Pseudo filesystems", and the configuration file. config generates the statement "Config_proc_vmcore=y".

The Dump Capture kernel configuration option (dependent on the processor architecture i386 and x86_64) is described below: On the processor architecture i386, open high-end memory support in the menu entry "Processor type and Features", configuration file. config Generation statement " Config_highmem64g=y "or" config_highmem4g ". On the processor architecture i386 and x86_64, turn off symmetric multiprocessor support in the menu entry "Rocessor type and Features", configuration file. config generates the statement "Config_smp=n". If the setting in the configuration file is "Config_smp=y", you can specify "Maxcpus=1" on the kernel command line of the mount dump capture kernel. If you want to build and use a relocatable kernel, open the option "build a relocatable kernel" in the menu entry "Rocessor type and Featuresif", configuration file. config statement "Config_ Relocatable=y ". In the menu under "Processor type and Features", the entry "Physical address where the kernel is loaded" sets the appropriate value for the physical addresses of the kernel mount. It appears only when the "kernel crash dumps" is turned on. The appropriate value depends on whether the kernel can be relocated.

If the value "config_physical_start=0x100000" is set, the relocatable kernel is used. It will compile the kernel at the physical address 1MB where the kernel is relocatable, so the kernel can run from any physical address. Kexec Bootloader will mount the kernel to the kernel retention area used to dump the capture kernel.

Otherwise, the start parameter "crashkernel=y@x" is used to specify that the second kernel retains the start address of the kernel zone, where Y represents the size of the memory area, and X represents the start address of the memory area reserved for the dump capture kernel, with x 16MB (0x1000000), so the user can set the "config_physical_start=0x1000000".

After the kernel is configured, the kernel and kernel modules are compiled and installed.

3) Extended Crashkernel syntax

In the boot command-line option of the system kernel, the usual syntax "crashkernel=size[@offset]" is sufficient for most configurations, but sometimes the reserved memory is dependent on system RAM. At this point, the memory can be limited by the extended Crashkernel command line to avoid removing a portion of the kernel from the machine and causing the system to not start. The extended Crashkernel syntax is listed as follows:

Crashkernel=<range1>:<size1>[,<range2>:<size2>,...] [@offset]

of which, Range=start-[end].

For example: crashkernel=512m-2g:64m,2g-:128m, meaning: If the memory is less than 512M, do not set the reserved memory, if the memory is between 512M to 2G, set reserved memory area for 64M, if the memory is greater than 128M, Set the reserved memory area to 128M.

4) booting into the system core

Update bootloader if necessary. The system kernel is then started with the parameter "crashkernel=y@x", such as: crashkernel=64m@16m, which tells the system kernel to keep the 64MB size from the physical address 0x01000000 (16MB) to the dump capture kernel for use. Usually the x86 and x86_64 platform Settings "crashkernel=64m@16m", PPC64 platform Settings "crashkernel=128m@32m".

5) Load dump capture kernel

After booting into the system kernel, you need to mount the dump capture kernel. Depending on the type of processor architecture and mapping file (relocatable), you can choose to mount uncompressed vmlinux or compressed bzimage/vmlinuz kernel images. The selection method description is as follows:

For i386 and x86_64 platforms: if the kernel is not relocatable, use Vmlinux. If the kernel is relocatable, use Bzimage/vmlinuz.

For PPC64 platforms: use Vmlinux.

For IA64 platforms: use Vmlinux or vmlinuz.gz. If the user uses an uncompressed vmlinux image, use the following command to mount the dump capture kernel:

Kexec-p <dump-capture-kernel-vmlinux-image> \
   --initrd=<initrd-for-dump-capture-kernel>-- Args-linux \
   --append= "root=<root-dev> <arch-specific-options>"

If the user is using a compressed bzimage/vmlinuz image, mount the dump capture kernel using the following command:

Kexec-p <dump-capture-kernel-bzimage>\

  --initrd=<initrd-for-dump-capture-kernel> \

   --append= "Root=<root-dev> <arch-specific-options>"

Note: The parameter--args-linux is not specified in the IA64 platform.

The following are the schema-specific command-line options used when loading dumps to capture the kernel: for i386, x86_64 and IA64 platforms, the option is "1 irqpoll maxcpus=1 reset_devices". For PPC64 platforms, the option is "1 Maxcpus=1 noirqdistrib reset_devices".

The things to note when loading a dump to capture the kernel are as follows: By default, elf headers are stored in ELF64 format to support systems with more than 4GB cores, on i386, Kexec automatically checks to see if physical RAM sizes exceed 4GB limits, and if not, use ELF32. Therefore, Elf headers always use the ELF32 format on non-PAE systems. Option--elf32-core-headers can be used to force the generation of ELF32 headers, which is necessary because GDB cannot currently open vmcore files with ELF64 headers on 32-bit systems. In the dump capture kernel, the startup parameter Irqpoll reduces driver initialization due to shared interrupts. The user must specify <root-dev> in the format of the root device name that the command mount outputs. The start parameter "1" launches the Dump capture kernel into Single-user mode that does not support the network. If the user wants to use the network, it needs to be set to 3. Typically, you do not have to let the dump capture kernel run in SMP mode. Therefore, you typically compile a single CPU dump to capture the kernel or mount the dump to capture the kernel with the specified option "Maxcpus=1".

6 trigger Kernel boot when kernel crashes

After the mount dump captures the kernel, if the system crashes (Kernel Panic), the system reboots into the dump capture kernel. The trigger point of the reboot is the function die (), the Die_nmi () and the SYSRQ processing routines (press the ALT-SYSRQ-C key combination).

The following conditions perform a crash trigger point: If a hardware lock is detected and "NMI watchdog" is configured, the system invokes the function Die_nmi () to boot into the dump capture kernel. If the function die () is invoked and the PID of the thread is 0 or 1, or if the die () is invoked in the interrupt context, or the panic_on_oops is set and the Die () is invoked, the system initiates the entry to the dump capture kernel. In the PowerPC system, when a soft reset is generated, all CPUs call Die (), and the system will boot into the dump capture kernel. For testing purposes, users can use "alt-sysrq-c", "Echo C >/proc/sysrq-trigger" to trigger a crash, or write a kernel module to force the kernel to crash.

7) Write the dump file

After the dump capture kernel is started, you can write the dump file using the following command:

Cp/proc/vmcore <dump-file>

The user can also access the dump memory as a device/dev/oldmem in a linear raw stream view, using the following command to create the device:

Mknod/dev/oldmem C 1 12

Using command dd to copy the specific portion of the dump memory, the commands for copying the entire memory are listed below:

DD If=/dev/oldmem of=oldmem.001

8) Dump file analysis

The user should reboot into a stable kernel before analyzing the dump image. The user can use GDB to make a limited analysis of the copied dump. You should add the-G option when compiling the vmlinux to generate the symbols for debugging, and then debug the vmlinux with the following command:

GDB Vmlinux <dump-file>

SYSRQ Magic Key printing kernel information

SysRq "Magic key Combination" is a group of keys, composed of "Alt+sysrq+[commandkey]" three keys on the keyboard, where Commandkey is an optional key. SYSRQ Magic Combination keys provide the ability to control kernel or print kernel information, depending on the key combinations. The functional description of the SYSRQ Magic key combination is shown in table 1. Table 1 Functional descriptions of SYSRQ key combinations

Key Name	Function description
B	Start immediately without synchronizing or uninstalling the hard drive.
C	To obtain a crash dump, perform a kexe reboot.