Debugging of Linux System kernel

Http://www.ibm.com/developerworks/cn/linux/l-kdb/index.html

This article will first introduce some kernel code monitoring and error tracing techniques on the Linux kernel, which vary depending on the required usage environment and usage methods, and then focus on the source-level debugging methods of the three Linux cores.

Debugging is an essential part of the software development process, and in the process of Linux kernel development, it is inevitable to face the problem of how to debug the kernel. However, developers of Linux systems are reluctant to include a debugger in the Linux kernel source tree for the sake of ensuring kernel code correctness. [1] They thought the debugger in the kernel would mislead the developer and introduce a bad fix. So debugging the Linux kernel has always been a tricky problem for kernel programmers, and the hard work of debugging is a significant feature of kernel-level development that distinguishes it from user-level development.

Despite the lack of a built-in approach to debugging the kernel, the Linux system has evolved to monitor kernel code and error tracking in the process of kernel development. At the same time, many patches came into being, and they added support for kernel debugging to the standard kernel. Although some of these patches are not recognized by the official Linux organization, they are truly functional and powerful. When debugging kernel issues, it is useful to use these tools and methods to track kernel execution and view their memory and data structures.

This article will first introduce some kernel code monitoring and error tracing techniques on the Linux kernel, which vary depending on the required usage environment and usage methods, and then focus on the source-level debugging methods of the three Linux cores.

1. Debugging technology of kernel level software of Linux system

PRINTK () is one of the most common techniques for debugging kernel code. By adding PRINTK () debug calls at a specific location in the kernel code, you can print the information you care about directly to the screen, allowing you to observe the execution path of the program and the variables, pointers, and other information that you care about. The Linux kernel debugger (Linux kernel debugger,kdb) is a patch for the Linux kernel that provides a way to check the kernel memory and data structure when the system is operational. Oops, KDB in the article Mastering the Linux Debugging technology has detailed introduction, we can refer to. Kprobes provides an interface that forcibly enters any kernel routine and collects information from the interrupt processor without interference. Using Kprobes, you can easily collect debug information such as processor registers and global data structures without having to compile and start the Linux kernel frequently, refer to using the Kprobes debug kernel for details.

The above describes the common techniques and methods for Linux kernel debugging and tracking. Of course, the kernel debugging and tracing methods are more than the above mentioned. One of the common features of these debugging techniques is that they are not able to provide the source code level of effective kernel debugging means, some can only be called error tracking technology, so these methods can only provide limited debugging capabilities. The following is a description of three practical kernel debugging methods for source code level.

Back to top of page

2. Build a Linux kernel debug environment using KGDB

KGDB provides a mechanism for debugging the Linux kernel using GDB. Using KGDB, you can set breakpoints in the kernel, check the value of variables, run a single step tracker, and so on, as you would debug a normal application. Using KGDB debugging requires two machines, one as a development machine (development machines), the other as the target machine, the two machines through the serial port or Ethernet port connection. The serial connector is a cable with a RS-232 interface, which is connected to the 3rd foot (RXD) at the 2nd foot (TXD) on both ends of the line, and the 7th foot (ground foot) is connected directly. During debugging, the debugged kernel runs on the target machine, and the GDB debugger runs on the development machine.

Currently, KGDB publishes debuggers that support i386, x86_64, 32-bit PPC, SPARC, and several other architectures. See Resources for KGDB patches [4].

2. 1 Debugging principle of KGDB

Installing the KGDB Debug environment requires a KGDB patch for the Linux kernel, and the functionality required for GDB remote debugging implemented by the patch includes 3 main parts of command handling, trap handling, and serial communication. The main purpose of the KGDB patch is to add a debug stub to the Linux kernel. The debug stub is a small piece of code in the Linux kernel that provides a medium between the development machine running GDB and the debugged kernel. GDB communicates with the debug stub through the GDB serial protocol. The GDB serial protocol is a message-based ASCII code protocol that contains a variety of DEBUG commands. When a breakpoint is set, KGDB is responsible for adding a trap instruction before the command that sets the breakpoint, and the control is transferred to the debug stub when the breakpoint is executed. At this point, the task of debugging the stub is to use the Remote Serial communication protocol to transfer the current environment to GDB and then accept the command from GDB. The GDB command tells the stub what to do next, and when the stub receives the command to continue, it resumes the running environment of the program, returning control of the CPU back to the kernel.

2. 2 installation and setup of KGDB

Below we will take the Linux 2.6.7 kernel as an example to detail the establishment process of the KGDB debugging environment.

2.2.1 Hardware and software preparation

The following hardware and software configurations are taken from the system configuration of the author's experiments:

The version of the Kgdb patch follows the naming pattern: Linux-a-kgdb-b, where A represents the kernel version number of Linux, and B is the version number of KGDB. As an example of the kgdb patch used in the experiment, the Linux kernel version is linux-2.6.7 and the patch version is kgdb-2.2.

After physically connecting the serial line, use the following command to test the serial connection between the two machines, the Stty command can set the serial port parameters:

Perform on development machine:

Stty ispeed 115200 ospeed 115200-f/dev/ttys0

Execute on target machine:

Stty ispeed 115200 ospeed 115200-f/dev/ttys0

Perform on developement machine:

echo Hello >/dev/ttys0

Execute on target machine:

Cat/dev/ttys0

If the serial connection is not a problem, the "Hello" will be displayed on the target machine's screen.

2. 2. 2 Installation and Configuration

Below we need to apply kgdb patch to the Linux kernel, set kernel options and compile the kernel. This information is relatively small, the author here gives a detailed introduction. The following work is done on the development machine (developement), taking the test environment described above as an example, and some specific steps may be made in the actual environment to make the appropriate changes:

I, configuration and compilation of the kernel

[[email protected] tmp]# TAR-JXVF Linux-2.6.7.tar.bz2[[email protected] tmp] #tar-JXVF linux-2.6.7-kgdb-2.2.tar.tar[[ Email protected] tmp] #cd inux-2.6.7

Follow the instructions given in the file Readme in the directory patch package to perform the patch for the corresponding architecture. As the experiment was done on the i386 architecture, only patches needed to be installed: Core-lite.patch, I386-lite.patch, 8250.patch, Eth.patch, Core.patch, I386.patch. When applying patch files, follow the order specified in the series files in the KGDB software package, or you may have unexpected problems. The Eth.patch file is the patch you need to use to select the Ethernet port for the debug connection

。

The commands to apply the patch are as follows:

[[email protected] tmp] #patch-p1 <.. /linux-2.6.7-kgdb-2.2/core-lite.patch

If the kernel is correct, then the patch should be applied without any problems (no *.rej files are generated). After you add a patch to the Linux kernel, you need to configure the kernel. The kernel configuration can be used in any way you choose to configure the Linux kernel.

[[email protected] tmp] #make menuconfig

Select the Kgdb debug item in the Kernel hacking option in the Kernel configuration menu, for example:

  [*] Kgdb:kernel debugging with remote GDB       Method for KGDB communication (kgdb:on generic serial port (8250))  --->    [*] kgdb:thread   Analysis [*] Kgdb:console messages through Gdb[[email protected] tmp] #make

Before compiling the kernel, note the optimization options in the Linux directory under Makefile, and the compilation of the default Linux kernel is at the-o2 level of optimization. At this level of optimization, the compiler will make changes to the order in which some code in the kernel is executed, so there is a case where the program runs inconsistent with the code sequence when debugging. You can change the-O2 option in makefile to-O, but not remove-O, or the compilation will be problematic. In order for the compiled kernel to have debug information, be careful to add the-G option when compiling the kernel.

However, configuring the system will automatically turn on the debug option when the Kernel debugging->compile the Kernel with Debug Info option is selected. In addition, after selecting "Kernel Debugging with remote GDB", the configuration system will automatically open the "Compile the kernel with debug info" option.

Once the kernel has been compiled, use the SCP command to copy the relevant files to the target machine (other network tools, such as RCP, can also be used).

[[email protected] tmp] #scp arch/i386/boot/bzimage [email protected]:/boot/vmlinuz-2.6.7-kgdb[[email protected] tmp]# SCP System.map [Email protected]:/boot/system.map-2.6.7-kgdb

If the system starts up so that some of the required device drivers are not compiled into the kernel, then you need to do the following:

[[email protected] tmp] #mkinitrd/boot/initrd-2.6.7-kgdb 2.6.7[[email protected] tmp] #scp initrd-2.6.7-kgdb [email Protected]:/boot/initrd-2.6.7-kgdb

II, kgdb start-up

After you copy the compiled kernel to the target machine, you need to configure the system bootstrapper to add the kernel boot options. The following is a description of the KGDB kernel boot parameters:

As described in the table, the boot parameters of the kernel have been different from the previous version after the KGDB 2.0 release. When using the Grub bootstrapper, use the KGDB parameter directly as the boot parameter of the kernel vmlinuz. A sample configuration of the bootstrapper is given below.

Title 2.6.7 kgdbroot (hd0,0) kernel/boot/vmlinuz-2.6.7-kgdb ro root=/dev/hda1 kgdbwait kgdb8250=1,115200

When using Lilo as the bootloader, you need to place the KGDB parameter in a statement that is decorated by append. The following is a sample configuration for using Lilo as a bootstrapper.

Image=/boot/vmlinuz-2.6.7-kgdblabel=kgdb    read-only    root=/dev/hda3append= "gdb gdbttys=1 gdbbaud=115200"

After saving the above configuration and restarting the computer and choosing to start the kernel with debugging information, the kernel will stop after a short run to create the init kernel thread, print out the following information, and wait for the connection of the development machine.

Waiting for connection from remote GDB ...

Execute on the development machine:

Gdbfile vmlinuxset Remotebaud 115200target remote/dev/ttys0

Where Vmlinux is a link to a compiled Linux kernel file in the source code directory, which is a kernel file that has not been compressed, and the GDB program obtains various symbolic address information from the file.

This establishes a connection to the Kgdb debug interface on the target machine. Once the connection is established, there is no difference between debugging in Linux and debugging a common application. You can interrupt the execution of the target machine by typing CTRL + C at any time to perform specific debugging tasks.

In versions prior to KGDB 2.0, the executable file Gdbstart was generated in the Arch/i386/kernel directory after the kernel was compiled. Copy the file to the/boot directory of the target machine without changing the kernel's boot configuration file and using the command directly:

[[email protected] boot] #gdbstart-S 115200-t/dev/ttys0

The debug connection between the development machine and the target machine can be established after the boot of the KGDB kernel is completed.

2. 2. 3 debugging via the network interface

The KGDB also supports the use of an Ethernet interface as a connection port for the debugger. When applying a patch package to the Linux kernel, you need to apply the Eth.patch patch file. When configuring the kernel, select the Kgdb debug entry in kernel Hacking, configure the Kgdb debug port for the Ethernet interface, for example:

[*] Kgdb:kernel debugging with remote Gdbmethod for KGDB communication (Kgdb:on Ethernet)  ---> () kgdb:on generic S Erial Port (8250) (X) Kgdb:on Ethernet

In addition, when using the Eth0 network port as the debug port, the Grub.list configuration is as follows:

Title 2.6.7 kgdbroot (hd0,0) kernel/boot/vmlinuz-2.6.7-kgdb ro root=/dev/hda1 kgdbwait [email protected]5.13/,@192.168. 6.13/

The other process is the same as when you use the serial port as the connection ports.

Note: Although you can use the Ethernet port as the Kgdb debug port, using the serial port as the connection port is more simple, the KGDB project group recommends using the serial port as the debug port.

2. 2. 4 Debugging method of the module

The debugging of kernel loadable modules has its particularity. Since the address of each segment in the kernel module is finalized when the module is loaded into the kernel, GDB of the develop machine cannot obtain various symbolic address information. So, one of the problems that needs to be solved when using the Kgdb debugging module is to get the final load address information of the Loadable module in some way, and add that information to the GDB environment.

I, kernel module debugging method in the Linux 2.4 kernel

In the linux2.4.x kernel, you can use the INSMOD-M command to output the load information for a module, for example:

[Email protected] tmp]# insmod-m Hello.ko >modaddr

View the module load information file modaddr as follows:

. This           00000060  c88d8000  2**2.text           00000035  c88d8060  2**2.rodata         00000069  c88d80a0  2**5.......data           00000000  c88d833c  2**2.bss            00000000  c88d833c  2**2 ...

In this information, we are concerned with only 4 segments of the address:. Text,. Rodata,. Data,. BSS. The above address information is added to GDB on the development machine, so that the module function can be tested.

(GDB) Add-symbol-file hello.o 0xc88d8060-s. Data 0xc88d80a0-s. Rodata 0xc88d80a0-s. BSS 0x c88d833c

This method also has some shortcomings, it can not debug the module initialization code, because the module initialization code has been executed. It is more impossible to set a breakpoint before the module is initialized without executing the module's load and not getting the module's insert address. The following workarounds can be used for this debug requirement.

Using the above method to get the address information of the module load on target machine, then unload the module with Rmmod. The resulting module address information is imported into the GDB environment on the development machine, and breakpoints are set before the kernel code calls the initialization code. This way, when the module is inserted again on the target machine, the code will stop before the module is initialized, so you can use the GDB command to debug the module initialization code.

Another way to debug the initialization function of a module is when inserting a kernel module, the kernel module mechanism calls the function Sys_init_module (KERNEL/MODLE.C) to initialize the kernel module, which invokes the initialization function of the inserted module. The program code snippet is as follows:

... if (mod->init! = NULL) ret = Mod->init ();..........

Setting breakpoints on this statement can also be stopped before the module initialization is executed.

II. Kernel module Debug method in Linux 2.6.x kernel

In the kernel after Linux 2.6, because of changes in the Module-init-tools tool, the INSMOD command no longer supports the-m parameter, and only takes other methods to get the address of the module loaded into the kernel. By analyzing the elf file format, we know that the meanings of the paragraphs in the program are as follows:

. Text (Code snippet): An operation instruction used to hold an executable file, that is, it is an image of a stored type that executes the program.

. Data segment: Data segments are used to store initialized global variables in an executable file, which is the variables and global variables that the program statically allocates.

. BSS (BSS segment): The BSS segment contains uninitialized global variables in the program, all zeros in the in-memory BSS segment.

. Rodata (Read-only segment): This segment holds read-only data and constructs a non-writable segment in the process image.

By placing the code in the module initialization function, we can easily get the address that the module loads into memory.

... int bss_var;static int hello_init (void) {PRINTK (kern_alert "text location. Text (Code Segment):%p\n", hello_init); static int DATA_VAR=0;PRINTK (kern_alert "data location. Data Segment):%p\n", &data_var);p rintk (Kern_alert " BSS location:. BSS (BSS Segment):%p\n ", &bss_var); ...} Module_init (Hello_init);

Here, by adding a simple program to the initialization function of the module, the module prints the loaded address in the kernel at load time. The address of the Rodata segment can be executed by executing the command readelf-e Hello.ko, get. Rodata the offset in the file and add the align value of the segment.

To enable the reader to better debug the module, the KGDB project also released a number of scripts to automatically detect the insertion of the module and automatically update the symbol information of the module in GDB. These scripts work in a similar way as previously explained, and for more information, read resources [4].

2. 2. 5 Hardware Breakpoints

The KGDB provides support for hardware debug registers. There are three hardware breakpoints that can be set in kgdb: Execution breakpoints (execution breakpoint), writing breakpoints (write breakpoint), Access breakpoints (Access breakpoint), and breakpoints that do not support I/O access. Currently, KGDB support for hardware breakpoints is implemented through macros, and you can set up to 4 hardware breakpoints, which are used as follows:

In some cases, the use of hardware breakpoints is very convenient for kernel debugging. For a definition of hardware breakpoints and specific instructions for use see resources [4]

。

2. 3. Build a debug environment in VMware

KGDB debugging environment needs to use two microcomputer to act as development machine and Target machine respectively, we can complete the KGDB debugging environment with only one computer after using VMware. Take the environment under Windows as an example, create two virtual machines, one as a development machine, and one as the target computer.

2. 3. 1 serial connection between virtual machines

Serial connections in virtual machines can take two methods. One is to specify the serial port of the virtual machine connected to the actual COM, for example, the development machine connected to the COM1, the target machine connected to the COM2, and then the two serial port through the serial cable connection. Another simpler approach is to support the mapping of serial ports to named pipes in some of the newer versions of VMware, and to map the serial ports of two virtual machines to the same named pipe. For example, in two virtual machines, select the same named pipe \\.\pipe\com_1, specify the COM port of the target machine as the server side, and choose the other end is a virtual machine property Specify the COM port of the development machine as the client side, and also specify the "The other end is a virtual machines" property of the COM port. For the IO Mode property, select the "Yield CPU on poll" complex selection box on target and the development machine is not selected. This allows the KGDB debug environment to be built using virtual machines without the need for any additional hardware. This reduces the hardware requirements for debugging with KGDB, and simplifies the process of setting up a debugging environment.

2. 3. 2 VMware tips for using

VMware virtual machines are relatively resource-intensive, especially if you use two virtual machines in Windows like the one above. Therefore, it is best to equip the system with more than 512M of memory and allocate at least 128M of memory per virtual machine. Such hardware requirements, for the current mainstream configuration of the PC is not too high requirements. For system performance, use the character interface as much as possible in VMware to debug your work. At the same time, the Linux system opens the SSHD service by default, it is recommended to use SECURECRT to log on to the Linux operation, so as to have a better user interface.

2. 3. 3 using KGDB in a virtual machine under Linux

For the use of VMware virtual machines under Linux, the author has not done any practical exploration. In principle, just to create a virtual machine under Linux as the target machine, the work of the development machine can be carried out in the actual Linux environment, the process of setting up the debugging environment is similar to the process described above. Because only one virtual machine needs to be created, using a virtual machine under Linux to build a KGDB debugging environment is less demanding for system performance. (VMware has launched the Linux version) can also be used on the development machine with some other debugging tools, such as more powerful cgdb, graphical interface of the DDD debugger, to facilitate the debugging of the kernel.

2. 4 some features and deficiencies of kgdb

The biggest disadvantage of using kgdb as the kernel debugging environment is that the requirements for the KGDB hardware environment are high, and two computers must be used as target and development machine respectively. Although the use of virtual machines can be used only a single PC can be set up debugging environment, but the performance of other aspects of the system also put forward certain requirements, but also increased the complexity of setting up the debugging environment. In addition, the KGDB kernel of the compilation, configuration is more complex, need a certain skill, the author at that time is also a lot of trouble. When the debugging process is finished, you will also need to re-create the kernel you want to publish. It is not possible to debug the whole process using kgdb, which means that kgdb is not used to debug the initial boot of the system.

However, Kgdb is a good kernel debugging tool that allows you to debug the kernel completely, and even debug the kernel's interrupt handlers. Debugging the kernel is more convenient with some graphical development tools.

Back to top of page

3. Build a Linux kernel debug environment using SkyEye

SkyEye is an open source software project (OpenSource software), and the goal of the SkyEye project is to emulate common embedded computer systems on common Linux and Windows platforms. SkyEye implements a command-level hardware simulation platform that simulates a variety of embedded development boards and supports a variety of CPU instruction sets. At the heart of SkyEye is the GNU GDB project, which combines GDB and ARM simulator nicely. With the addition of the Armulator feature, it can be used to emulate the embedded Development Board, which can debug not only the hardware driver, but also the operating system. SkyEye project has been widely promoted in the field of embedded system development.

3. 1 SkyEye Installation and Clinux kernel compilation

3. Installation of 1.1 SkyEye

SkyEye installation is not the focus of this article to introduce, there is a large number of information on this is introduced. For information on the installation and use of SkyEye, please refer to reference material [11]. Because the SkyEye face is mainly used in the field of embedded systems, the Clinux system is often used on skyeye, but it is also possible to use Linux as a system to run on SkyEye. Since the introduction of Clinux 2.6 on the SkyEye compilation of relevant data is not many, so the following detailed introduction.

3. Compilation of 1.2μclinux 2.6.x

To debug the operating system kernel in SkyEye, you must first enable the Debugged kernel to function correctly on the board that the SkyEye simulates. Therefore, correctly compiling the operating system kernel to be debugged and configuring SkyEye is the first step in kernel debugging. Below we use SkyEye to simulate the Atmel AT91X40-based Development Board, and run Clinux 2.6 as an example to introduce the specific debugging methods of SkyEye.

I, install the cross-compilation environment

Install the cross-compiler first. Although it is stated in some materials that the tool chain arm-elf-tools-20040427.sh is used, due to the difference between arm-elf-xxx and ARM-LINUX-XXX for macro and link processing, Experience proves that using the ARM-ELF-XXX tool chain will make an error in the last phase of link vmlinux. So here we use the cross-compiler toolchain: arm-uclinux-tools-base-gcc3.4.0-20040713.sh, see [6] for the cross-compiler toolchain. Note the following steps are best performed with the root user.

[[email protected] tmp] #chmod +x  arm-uclinux-tools-base-gcc3.4.0-20040713.sh[[email protected] tmp]#./ arm-uclinux-tools-base-gcc3.4.0-20040713.sh

After you install the cross-compilation toolchain, make sure that the tool chain installation path exists in the system path variable.

II. Making the Clinux kernel

One of the easiest ways to get a Clinux release package is to visit the uclinux.org site directly [7]. The kernel version released by this site may not be up-to-date, but you can find the latest Clinux patch and find a corresponding Linux kernel version to create the latest Clinux kernel. Here, you will use this method to make the latest clinux kernel. Currently (when I write this article), the latest version of the available release package is uclinux-dist.20041215.tar.gz.

Download uclinux-dist.20041215.tar.gz, see [7] for files.

Download linux-2.6.9-hsc0.patch.gz, see [8] for files.

Download linux-2.6.9.tar.bz2, see [9] for files.

Now we have the entire linux-2.6.9 source code, as well as the required kernel patches. Please prepare a directory with 2GB of space to complete the following process of making clinux kernel.

[Email protected] tmp]# TAR-JXVF uclinux-dist-20041215.tar.bz2[[email protected] uclinux-dist]# TAR-JXVF  Linux-2.6.9.tar.bz2[[email protected] uclinux-dist]# GZIP-DC linux-2.6.9-hsc0.patch.gz | Patch-p0

or use:

[Email protected] uclinux-dist]# gunzip linux-2.6.9-hsc0.patch.gz [[email protected] Uclinux-dist]patch-p0 < Linux-2.6.9-hsc0.patch

After performing the above procedure, a patch directory-armnommu will be generated under the Linux-2.6.9/arch directory. Remove the linux-2.6.x from the original Clinux directory (that is, the linux-2.6.9-uc0) and rename our patched Linux kernel directory to linux-2.6.x.

[Email protected] uclinux-dist]# RM-RF linux-2.6.x/[[email protected] uclinux-dist]# mv linux-2.6.9 linux-2.6.x

III. Configuring and compiling the Clinux kernel

Because only for the purpose of debugging the Clinux kernel, there are no uclibc library files and romfs.img files generated here. When publishing Clinux, some of the configuration files for some commonly used embedded boards have been provisioned, so the configuration files are used directly here, as follows:

[[Email protected] uclinux-dist]# CD Linux-2.6.x[[email protected] linux-2.6.x] #make Arch=armnommu cross_compile= Arm-uclinux-atmel_deconfig

The Atmel_deconfig file is a configuration file that is provided at Clinux release and is stored in the directory linux-2.6.x/arch/armnommu/configs/.

[[email protected] linux-2.6.x] #make Arch=armnommu cross_compile=arm-uclinux-oldconfig

Compile the configured kernel below:

[email protected] linux-2.6.x]# make Arch=armnommu cross_compile=arm-uclinux-v=1

In general, the compilation will end successfully and generate an uncompressed clinux kernel file Vmlinux in the linux-2.6.x/directory. It is important to note that in order to debug the Clinux kernel, you need to open the kernel-compiled debug option-G, so that the compiled kernel has debug information. To open the compile option, you can choose:

"Kernel debugging->compile the Kernel with debug info" will automatically turn on the debug option. You can also directly modify the makefile file in the linux-2.6.x directory to open the debug switch for it. The method is as follows:.

CFLAGS  + =-G

The most likely problem is the failure to find the ARM-UCLINUX-GCC command, mainly because the path variable does not contain the directory where the ARM-UCLINUX-GCC command resides. In the default installation of ARM-LINUX-GCC, its installation directory is/root/bin/arm-linux-tool/, using the following command to add paths to the PATH environment variable.

Export path= $PATH:/root/bin/arm-linux-tool/bin

IV, the creation of the root file system

One of the last operations of the Linux kernel at startup is to load the root file system. The root file system contains all the applications, library files, and other services that are used by the embedded system. For the sake of the length of the article, it is not intended to introduce the method of making the root file system, and readers can consult some other relevant information. It is important to note that the root file system mounted to the kernel is specified by the configuration file skyeye.conf.

3. 2 Debugging with SkyEye

After compiling the Clinux kernel, you can debug the elf execution file format kernel in SkyEye. As mentioned earlier, using the SkyEye debug kernel is the same as using GDB to debug a program.

It is necessary to remind the reader that the SkyEye configuration file-skyeye.conf records the simulated hardware configuration and the simulated execution behavior. This profile is one of the most important files in the SkyEye system, and many errors and anomalies occur in relation to the file. When there is an error installing configuration SkyEye, check the configuration file first and then do other work. At this point, all the preparation has been completed, you can do the kernel debugging work.

3. 3 features and disadvantages of using SkyEye to debug the kernel

The Linux system kernel can be debugged in SkyEye. Because SkyEye currently supports the arm core-based CPUs, it is generally necessary to compile the Linux system kernel to be debugged using the cross-compilation tool. In addition, the kernel compiling and configuration process used in making skyeye is more complicated and cumbersome. However, you do not need to re-create the kernel you want to publish after the debugging process is finished.

SkyEye only to a certain degree of system hardware simulation, so in the SkyEye and real hardware environment compared to a certain gap, which is closely related to hardware debugging may have some impact, such as driver debugging. For most software debugging, however, SkyEye has provided enough simulation for accuracy.

SkyEye's next goal is to integrate with Eclipse, with a graphical interface that provides some convenience for debugging and viewing the source code.

Back to top of page

4. Debugging the Linux kernel with UML

User-mode Linux (UML) is basically Linux running inside Linux. The project is to make the Linux kernel a separate, user-space process running on Linux systems. Instead of running on a new hardware architecture, UML runs on virtual machines implemented in a Linux-based system invocation interface. Because UML is a feature that runs Linux as a user-space process, you can use UML to debug the kernel of the operating system. For an introduction to UML, refer to resources [10], [12].

4. 1 installation and commissioning of UML

The installation of UML requires a running Linux 2.2.15 or more, or 2.3.22 or more I386 machines. For 2.6.8 and its previous versions of UML, it is published in two forms: one is released in the form of RPM packages, and the other is the installation of UML in the form of a source code. As described in UML, the installation packages provided in RPM are obsolete and have many problems. A UML package that is published in binary form does not contain the required debugging information, which is optimized for a different degree when published. So, to debug a Linux system kernel with UML, you need to compile and install UML with the latest UML patch code and the corresponding version of the Linux kernel. After completing the UML patch, a um directory is generated in the Arch directory, and the main UML code is placed in that directory.

Since the 2.6.9 release (including 2.6.9 version of Linux), User-mode Linux has been released with the Linux kernel source tree, which is stored in the Arch/um directory.

Once the UML kernel has been compiled, it can be debugged by running the compiled kernel directly using GDB.

4. 2 debugging system kernel features and weaknesses using UML

At present, the user mode Linux virtual machine also has some limitations. Because a UML virtual machine is a virtual machine implemented based on a Linux system invocation interface, the user-mode kernel cannot access the hardware devices on the host system. Therefore, UML is not suitable for debugging drivers that handle actual hardware. However, if you are writing kernel programs that are not hardware-driven, such as Linux file systems, protocol stacks, and so on, using UML as a debugging tool is a good choice.

Back to top of page

5. Kernel Debug Configuration options

To facilitate debugging and testing of code, the kernel provides many configuration options related to kernel debugging. Most of these options are in the kernel development (kernel hacking) menu item in the kernel configuration Editor. There are also configurable debugging options elsewhere in the kernel Configuration Tree menu, which will be described below.

Page alloc Debugging:config_debug_pagealloc:

When this option is not used, the freed pages of memory are removed from the kernel address space. With this option, the kernel defers the process of moving out of memory pages, so it can discover errors in memory leaks.

Debug Memory Allocations:config_debug_slab:

When this option is turned on, several types of checks are performed before the kernel performs memory allocations, and these types of checks can detect errors such as kernel oversubscription or uninitialized initialization. The kernel will set some alert values before and after each allocation of memory, and if the values change then the kernel will know that the memory has been manipulated and given explicit hints, making the various cryptic errors easier to trace.

Spinlock Debugging:config_debug_spinlock:

When this option is turned on, the kernel will be able to discover spinlock uninitialized and various other errors that can be used to troubleshoot some of the deadlock-induced errors.

Sleep-inside-spinlock Checking:config_debug_spinlock_sleep:

When this option is turned on, the owner of the Spinlock will perform the appropriate checks when it sleeps. In fact, even if the caller does not have sleep at the moment, there is a hint of the possibility of sleep.

Compile the kernel with debug Info:config_debug_info:

When this option is turned on, the compiled kernel will contain all debugging information, which is required when using GDB.

Stack Utilization Instrumentation:config_debug_stack_usage:

This option is used to track overflow errors in the kernel stack, and the obvious symptom of a kernel stack overflow error is that it generates a oops error without listing the system's call stack information. This option causes the kernel to check for stack overflow and make the kernel count the stack usage.

Driver Core verbose debug Messages:config_debug_driver:

This option is located under Device drivers-> Generic Driver options, which enables the kernel driver core to generate a lot of debugging information and log them to the system log.

Verbose SCSI Error Reporting (kernel size +=12k): config_scsi_constants:

This option is located under Device DRIVERS/SCSI device support. When a SCSI device fails, the kernel gives a detailed error message.

Event Debugging:config_input_evbug:

When this option is turned on, errors and all events from the input subsystem are output to the system log. This option results in a detailed input report, as well as a certain security issue.

The above kernel compilation options require the reader to choose flexibly according to the actual situation of the kernel programming. When using the three source code level kernel Debugging tools described above, it is generally necessary to select the Config_debug_info option so that the compiled kernel contains debug information.

Back to top of page

6. Summary

The above describes some methods of debugging the Linux kernel, especially the three source code-level kernel debugging tools, and the way to build these kernel debugging environment, readers can choose according to their own situation.

Debugging tools, such as GDB, require the support of the operating system, while the kernel cannot properly perform the administrative functions of the system due to some faulty code, so debugging the kernel must take some special methods. The above introduction of the three source code-level debugging methods, can be summarized as the following two strategies:

I, for the kernel to add debugging stubs, the use of debugging stubs for remote debugging, this debugging strategy requires target and development machine to complete the debugging task.

II, the combination of virtual machine technology and debugging tools, so that the Linux kernel run in the virtual machine to use the debugger to debug the kernel. This strategy requires the creation of a system kernel that is suitable for running in a virtual machine.

Different debugging strategies determine the different working principles of debugging, and also form various debugging methods of different hardware and software requirements and their respective characteristics.

In addition, it is important to note that the mastery of the kernel debugging capability depends largely on experience and in-depth understanding of the entire operating system. A thorough and thorough understanding of the kernel of the system will greatly accelerate the development and commissioning of the Linux system kernel.

The debugging techniques and methods for the kernel of the system are not the only ones covered above, here are just a few ways to see and hear. While the Linux kernel is moving forward, the kernel's debugging techniques are constantly progressing. It is hoped that some of the methods described above will help readers develop and learn Linux.

Debugging of Linux System kernel