Secrets of Linux kernel debugger

Guidance:
　　Secrets of Linux kernel debugger
KDB Getting Started Guide
It is useful to track kernel execution and view its memory and data structure when debugging kernel problems. The built-in inner-core debugger KDB in Linux provides this function. In this article, you will learn how to use the features provided by KDB and how to install and set KDB on Linux machines. You will also be familiar with the commands that can be used in KDB and the settings and display options.
The Linux kernel debugger (KDB) allows you to debug the Linux kernel. This tool, like its name, is essentially a patch of kernel code, which allows experts to access the kernel memory and data structure. One of the main advantages of KDB is that it does not need to be debugged on another machine: You can debug the running kernel.
It takes some work to set a machine for KDB, because the kernel needs to be patched and re-compiled. KDB users should be familiar with the compilation of the Linux kernel (to a certain extent, they must be familiar with the internal mechanism of the kernel). However, if you need to compile the kernel, see references at the end of this article.
In this article, we will start with information on downloading KDB patches, patching, (re-Compiling) the kernel, and starting KDB. Then we will understand the KDB commands and study some common commands. Finally, we will look at some details about settings and display options.
Getting started
The KDB project is maintained by Silicon Graphics (see references for links). You need to download patches related to the kernel version from its FTP site. (When writing this article) the latest available KDB version is 4.2. You will need to download and apply two patches.
One is a "public" patch that includes changes to the general kernel code, and the other is a system-specific patch. The patch can be obtained as an bz2 file. For example, on an x86 machine running the 2.4.20 kernel, you need kdb-v4.2-2.4.20-common-1.bz2 and kdb-v4.2-2.4.20-i386-1.bz2.
All examples provided here are for the i386 architecture and the 2.4.20 kernel. You will need to make appropriate changes based on your machine and kernel version. You also need the root permission to perform these operations.
Copy the file to the/usr/src/Linux directory and extract the patch file from the file compressed with Bzip2:
# Bzip2-D kdb-v4.2-2.4.20-common-1.bz2
# Bzip2-D kdb-v4.2-2.4.20-i386-1.bz2
You will get kdb-v4.2-2.4.20-common-1 and kdb-v4.2-2.4-i386-1 files.
Apply these patches:
# Patch-p1
These patches should be applied cleanly. Search for any file ending with. rej. This extension indicates that these are failed patches. If there is no problem with the kernel tree, the patch application will not have any problems.
Next, you need to build a kernel to support KDB. The first step is to set the config_kdb option. Use your favorite configuration mechanisms (such as xconfig and menuconfig) to complete this step. Go to the "kernel Hacking" section at the end and select the "Built-in kernel debugger support" option.
You can also select the other two options based on your preferences. Select the "compile the kernel with frame Pointers" option (if any), set the config_frame_pointer flag. This produces better stack backtracking because frame pointer registers are used as frame pointers rather than general registers.
You can also select the "KDB off by default" option. This sets the config_kdb_off flag, and KDB is disabled by default. We will introduce this in detail in later sections.
Save the configuration and exit. Recompile the kernel. We recommend that you run "make clean" before building the kernel ". Install and boot the kernel in common ways.
Initialize and set Environment Variables
You can define the KDB command that will be executed during KDB initialization. You need to define these commands in the plain text file kdb_cmds, which is located in the KDB directory of the Linux source code tree (after patches are installed, of course. This file can also be used to define environment variables for setting display and Print Options. Annotations at the beginning of the file provide help in editing the file. The disadvantage of using this file is that you need to re-build and re-install the kernel after you change the file.
Activate KDB
If config_kdb_off is not selected during compilation, KDB is active by default. Otherwise, you need to activate it explicitly-pass the KDB = on flag to the kernel during boot or execute the job after/proc is mounted:
# Echo "1">/proc/sys/kernel/KDB
If you perform the preceding steps, KDB is deactivated. That is to say, if KDB is enabled by default, The KDB = off flag is passed to the kernel or the following operation will cancel KDB activation:
# Echo "0">/proc/sys/kernel/KDB
Another flag can be passed to the kernel during boot. The KDB = early flag will cause control to be passed to KDB at the initial stage of the boot process. This is helpful if you need to debug in the initial phase of the boot process.
There are many ways to call KDB. If KDB is on, it is automatically called as long as there is an emergency in the kernel. Press the pause key on the keyboard to manually call KDB. Another way to call KDB is through the serial console.
To do this, you need to set up the serial console (see references for help) and a program to read data from the serial console. Press ctrl-a to call KDB from the serial console.
KDB command
KDB is a powerful tool that allows several operations, such as memory and register modification, application breakpoint, and stack tracing. Based on these, KDB commands can be divided into several categories. The following describes the most common commands in each type.
Memory display and Modification
The most common commands in this category are MD, MCM, mm, and MMW.
The MD command takes an address/symbol and a row count as the parameter to display the line-count row memory starting from this address. If line-count is not specified, the default value specified by the environment variable is used. If no address is specified, the MD continues from the last printed Address. The address is printed at the beginning, and the character conversion is printed at the end.
The MCM command carries the address/symbol and the byte count to display the initial memory content of the byte-count bytes starting from the specified address. It is essentially the same as MD, but it does not show the starting address and does not show character conversion at the end. The command is rarely used.
MM command to modify the memory content. It uses the address/Symbol and new content as parameters and replaces the content at the address with new-contents.
The MMW command changes the W bytes starting from the address. Note that mm must change the machine word.
Example
Display the 15 rows of memory starting from 0xc000000:
0] KDB> MD 0xc000000 15
Change the content at the memory location 0xc000000 to 0x10:
0] KDB> MM 0xc000000 0x10
Display and modify registers
Commands in this category include Rd, RM, and EF.
The RD command (without any parameters) displays the content of the Processor register. It can have three parameters. If the C parameter is passed, RD displays the control register of the processor; if it carries the D parameter, it displays the debugging register; if it carries the U parameter, displays the register group of the last task that entered the kernel.
Rm command to modify the register content. It uses the register name and new-contents as parameters and uses new-contents to modify the register. The register name is related to a specific architecture. Currently, the control register cannot be modified.
The EF command uses an address as a parameter to display abnormal frames at a specified address.

Example

Display general register group:

??? 0] KDB> Rd

Set the content of the register EBX to 0x25:

??? 0] KDB> RM % EBX 0x25

Breakpoint

Common breakpoint Commands include bp, BC, BD, Be, and Bl.

The BP command uses an address/symbol as a parameter. It applies a breakpoint at the address. When this breakpoint occurs, the execution is stopped and the control is handed over to KDB. This command has several useful variants. The BPA command applies breakpoints to all processors in the SMP system. The ALB Command forces it to be used on a system that supports hardware registers. The BPHA command is similar to the BPA command, except that it forces the use of hardware registers.

The BD command disables special breakpoints. It receives the breakpoint number as a parameter. This command does not remove a breakpoint from the breakpoint table, but simply disables it. The breakpoint number starts from 0 and is allocated to the breakpoint according to the availability order.

Be command to enable breakpoint. The parameter of this command is also the breakpoint number.

The BL command is used to list the current breakpoint set. It contains enabled and disabled breakpoints.

The BC command removes breakpoints from the breakpoint table. It uses a specific breakpoint number or * as a parameter. In the latter case, it removes all breakpoints.

For example, set a breakpoint for the function sys_write:

??? 0] KDB> BP sys_write

List all breakpoints in the breakpoint table:

??? 0] KDB> BL

Clear breakpoint 1:

??? 0] KDB> BC 1

Stack Tracing

Major Stack trace Commands include BT, BTP, BTC, and BTA.

The BT command tries to provide information about the stack of the current thread. It can selectively use the stack frame address as a parameter. If the address is not provided, it uses the current Register to trace the stack. Otherwise, it assumes that the provided address is a valid starting address of the stack frame and tries to trace back. If the config_frame_pointer option is set during kernel compilation, the frame pointer register is used to maintain the stack, so that stack tracing can be correctly executed. If config_frame_pointer is not set, the BT command may produce incorrect results.

The BTP command uses the process ID as a parameter and traces back the stack of the specific process.

The BTC command executes stack backtracking for processes running on each active CPU. It starts to execute BT from the first active CPU, switches to the next active CPU, and so on.

The BTA command traces back all processes in a specific State. If no parameter is included, it will trace back all processes. You can selectively pass parameters to the command. Processes in a specific State are processed based on parameters. Options and the corresponding status are as follows:

D: Non-disruptive status

R: Running

S: resumable sleep

T: tracked or stopped

Z: Dead

U: not running

Each of these commands prints a large amount of information. For more information about these fields, see the following references.

Example

Stack that traces the current active thread:

 

??? 0] KDB> BT

Stack of processes marked as 575:

??? 0] KDB> BTP 575

Other commands

The following are several other KDB commands that are useful during kernel debugging.

The id command uses an address/symbol as a parameter. It disassembles the commands starting from this address. The environment variable idcount determines the number of rows to be displayed.

The SS Command executes the command in one step and then returns the control to KDB. One variant of this command is SSB, which executes commands starting from the current Instruction Pointer address (printing commands on the screen) until it encounters commands that will cause branch transfer. Typical examples of branching Commands include call, return, and jump.

The go command allows the system to continue normal execution. Run until a breakpoint is reached (if a breakpoint has been applied ).

The reboot command immediately restarts the system. It does not completely shut down the system, so the results are unpredictable.

The ll command takes the address, offset, and another KDB command as the parameter. It repeatedly executes this command as a parameter for each element in the linked list. The executed command takes the address of the current element in the list as the parameter.

Example

Disassembles commands starting with schedule. The number of lines displayed depends on the environment variable idcount:

??? 0] KDB> SSB

??? 0xc0105355 default_idle + 0x25: CLI
??? 0xc0105356 default_idle + 0x26: mov 0x14 (% EDX), % eax
??? 0xc0105359 default_idle + 0x29: Test % eax, % eax
??? 0xc010535b default_idle + 0x2b: JNE 0xc0105361 default_idle + 0x31

Tips and tricks

Debugging a problem involves using a debugger (or any other tool) to find the root cause of the problem and using source code to track the root cause of the problem. It is extremely difficult to identify the problem by using the source code. Only sophisticated kernel hackers can do this. On the contrary, most new users often rely too much on debuggers to correct errors.

This method may produce incorrect problem solutions. We are worried that this method will only correct the surface symptoms and cannot solve the real problem. A typical example of this type of error is to add the error processing code to handle null pointers or error references, but the real cause of the invalid reference is not found.

Combining research code with debugging tools is the best solution to identify and correct problems.

The main purpose of the debugger is to locate the wrong location, confirm the symptoms (in some cases there is a cause), determine the value of the variable, and determine how the program appears in this situation (that is, create a call stack ). Experienced hackers will know which debugger should be used for a specific problem, and can quickly obtain necessary information based on debugging, and then continue to analyze the code to identify the cause.

Therefore, here are some tips for you to use KDB to quickly obtain the above results. Of course, remember that the speed and accuracy of debugging come from experience, practice, and good system knowledge (hardware and internal kernel mechanisms ).

Tip 1

In KDB, enter the address at the prompt to return the most matched symbol. This is extremely useful in stack analysis and the determination of global data addresses/Values and function addresses. Similarly, if you enter a symbolic name, the virtual address is returned.

Example

It indicates that the function sys_read starts from the address 0xc013db4c:

[0] KDB> 0xc013db4c
0xc013db4c = 0xc013db4c (sys_read)

Similarly, sys_write is at address 0xc013dcc8:

[0] KDB> sys_write
Sys_write = 0xc013dcc8 (sys_write)

These help to locate global data and function addresses when analyzing the stack.

Tip 2

When compiling a kernel with KDB, use this option as long as the config_frame_pointer option appears. Therefore, you need to select the "compile the kernel with frame Pointers" option under the "kernel Hacking" section When configuring the kernel.

This ensures that the frame pointer register will be used as the frame pointer to generate correct backtracking. In fact, you can manually dump the content of the frame pointer register and track the entire stack. For example, on the i386 machine, the % EBP register can be used to trace the entire stack.

For example, after executing the first command on the rmqueue () function, the stack looks similar to the following:

 

 

??? 0] KDB> MD % EBP

??? 0xc74c9f38 c74c9f60 c0136c40 000001f0 00000000
??? 0xc74c9f48 08053328 c00000238 c000003a8 00000000
??? 0xc74c9f58 000001f0 00000246 c74c9f6c c0136a25
??? 0xc74c9f68 c74c8000 c74c9f74 c0136d6d c74c9fbc
??? 0xc74c9f78 c014fe45 c74c8000 00000000 08053328

??? 0] KDB> 0xc0136c40

??? 0xc0136c40 = 0xc0136c40 (_ alloc_pages + 0x44)

??? 0] KDB> 0xc0136a25

??? 0xc0136a25 = 0xc0136a25 (_ alloc_pages + 0x19)

??? 0] KDB> 0xc0136d6d

??? 0xc0136d6d = 0xc0136d6d (_ get_free_pages + 0xd)

We can see that rmqueue () is called by _ alloc_pages, and the latter is called by _ alloc_pages, and so on.

The first double word in each frame points to the next frame, which is followed by the address for calling the function. Therefore, the tracing stack becomes an easy task.

Tip 3

The go command can selectively use an address as a parameter. If you want to continue execution at a specific address, you can provide this address as a parameter. Another way is to use the RM command to modify the instruction pointer register and then enter go. This is useful if you want to skip a specific command or a group of commands that seem to cause problems. However, please note that this command may cause serious problems and the system may crash.
 

Tip 4

You can use a command named defcmd to define your own command set. For example, whenever a breakpoint occurs, you may want to check a special variable, check the content of some registers, and dump the stack. Generally, you must enter a series of commands to execute all these tasks at the same time.

Defcmd allows you to define your own commands. This command can contain one or more predefined KDB commands. Then, you only need to use one command to complete all the three tasks. The syntax is as follows:

 

??? 0] KDB> defcmd name "usage" "Help"

??? 0] KDB> [defcmd] type the commands here

??? 0] KDB> [defcmd] endefcmd

For example, you can define a (simple) new command, Hari, which displays a line of memory starting from the address 0xc000000, displays the register content, and dumps the stack:

??? 0] KDB> defcmd Hari "" No arguments needed"

??? 0] KDB> [defcmd] MD 0xc000000 1

??? 0] KDB> [defcmd] Rd

??? 0] KDB> [defcmd] MD % EBP 1

??? 0] KDB> [defcmd] endefcmd

The command output will be:

??? 0] KDB> Hari

??? Hari] KDB> MD 0xc000000 1

??? 0xc000000 00000001 f000e816 f000e2c3 f000e816

??? Hari] KDB> Rd

??? Ax = 0x00000000 EBX = 0xc0105330 ECx = 0xc0466000 edX = 0xc0466000
???....
???...

??? Hari] KDB> MD % EBP 1

??? 0xc0467fbc c0467fd0 c01053d2 00000002 000a0200

??? 0] KDB>

Tip 5

You can use the following commands (if the architecture supports using hardware registers) to apply read/write breakpoints. This means that we can control this whenever we read data from a specific address or write data to this address. This can be extremely convenient when debugging data/memory corruption occurs. In this case, you can use it to identify corrupted code/processes.

Example

Enter the kernel debugger whenever four bytes are written to the address 0xc0204060:

??? 0] KDB> 0x60 dataw 4

When reading data of at least two bytes starting from 0xc000000, enter the kernel debugger:

??? 0] KDB> 0x2

Conclusion

For kernel debugging, KDB is a convenient and powerful tool. It provides various options and enables us to analyze memory content and data structures. The best thing is that it does not need to use another machine for debugging.

This article is transferred from
Http://forums.cweek.com.cn/thread-54176-1-1.html