Kernel Debugging Tools-kdump & Crash

Kdump Introduction

Kdump is a tool used to dump running memory when the system crashes.

Once the system crashes, the kernel will not work properly, and this time the kdump will provide a kernel to capture the current running information.

The kernel collects all running state and data information in memory at this time into a dump core file for later analysis of the cause of the crash.

Once the memory information is collected, you can have the system restart automatically.

Kdump is supported after RHEL5, 2006 is accepted as part of the kernel by the mainline. It's a simple principle to keep a piece in memory

Area, this area is used for storing capture kernel, when the production kernel occurs crash, the reserved area by kexec

Capure kernel to run, and the capture kernel is responsible for the complete information of the product kernel-including CPU registers, stack data, and so on dumps

To a file in the specified location.

Kdump principle

Kexec is the key to the kdump mechanism and consists of two parts:

System call Kexec_load in kernel space. Responsible for loading the capture kernel to the specified address when the production kernel is started.

Tool kexec-tools for user space. The address of the capture kernel is passed to the production kernel, which finds the address of the capture kernel and runs when the system crashes.

Kdump is a kernel crash dump mechanism based on kexec. When the system crashes, Kdump uses kexec to boot to the second kernel. A second kernel typically

Called the capture kernel, which starts with very little memory to capture the dump image. The first kernel retains part of the memory for the second kernel to start using.

Because Kdump uses kexec to boot the capture kernel, bypassing the BIOS, the first kernel memory is retained. This is the nature of the memory crash dump.

After the capture kernel starts, it runs like a normal kernel to run the INIT program on the RAMDisk created for it. And all kinds of dump mechanism can be implemented in Init in advance.

In order to successfully start the capture kernel when the production kernel crashes, the capture kernel and its ramdisk are placed in the memory of the production kernel beforehand.

The memory of the production kernel is delivered to the capture kernel by/proc/vmcore this file. To generate it, the user tool analyzes the use of memory in the production kernel and

Distribution, and then combine this information to generate an Elf header file to save. When the capture kernel is booted, it is delivered simultaneously by the Elf file header.

Address, by analyzing it, capturing the kernel can generate a/proc/vmcore. With the/proc/vmcore file, the script in the capture kernel's RAMDisk

A variety of strategies can be implemented through common file reading and writing and networking.

Kdump Configuration

RHEL5 started, Kexec-tools is installed by default.

If you need to debug a kdump-generated vmcore file, you need to manually install the Kernel-debuginfo package.

(1) Reserved memory

You can modify the kernel boot parameters to reserve the specified memory for the start capture kernel.

In/etc/grub.conf (typically a soft link for/boot/grub/grub.conf):

Crashkernel=y@x,y is the memory that the kernel retains for the Kdump capture, and X is the starting position for reserving part of the memory.

The default is Crashkernel=auto, which can be set as crashkernel=256m.

(2) configuration file

The configuration file is/etc/kdump.conf and the following are several common configurations:

# Path/var/crash

The default Vmcore storage directory is/var/crash/%host-%date/, including two files: Vmcore and Vmcore-dmesg.txt

# SSH <user@service>

Would copy/proc/vmcore to <user@server>:<path>/%host-%date/via SSH

Make sure user has necessary write permissions on server.

Automatically copy to the remote machine.

# Default <reboot | Halt | Poweroff | Shell | Mount_root_run_init>

The Actions to perform in the case dumping to intended target fails.

Execution occurs when a dump fails.

(3) Start service

# chkconfig Kdump on//boot

# service Kdump status/Start, stop, restart, etc.

(4) Functional verification

Magic System Request key is a magical key combo can hit which the kernel'll respond to regardless

Of whatever else it is doing, unless it are completely locked up.

Using SYSRQ requires the support of compilation options CONFIG_MAGIC_SYSRQ. More information can be seen documentation/sysrq.txt.

Deliberately let the system crash to test whether Kdump is working properly.

# echo C >/proc/sysrq-trigger

Would perform a system crash by a NULL pointer dereference.

A crash dump would be taken if configured.

Magic SysRq also has some very interesting values, some of which have a great break-through, output in/var/log/messages:

F:call Oom_kill to kill a memory hog process. Perform Oom killer.

L:shows a stack backtrace for all active CPUs. Print out the stack backtrace for all CPUs.

M:dump Current Memory info. Print out memory usage information.

P:dump the current registers and flags. Prints the register information for the CPU on which it resides.

(5) Capturing the kernel

The capture kernel is an uncompressed elf image file to see if the capture kernel is loaded into memory:

# cat/sys/kernel/kexec_crash_loaded

To reduce the memory consumed by the capture kernel:

# echo N >/sys/kernel/kexec_crash_size

Crash Introduction

When the system crashes, through Kdump can get the memory dump file Vmcore, but how to analyze vmcore it.

Crash is a tool for analyzing kernel dump files, used in general and kdump.

When using crash, you are required to debug the kernel vmlinux with the-G option at compile time, with debugging information.

If Vmcore is not specified, the memory of the real-time system is used to parse by default.

It is worth mentioning that crash can also be used to analyze real-time system memory, is a very powerful debugging tool.

Crash uses GDB as an internal engine, the syntax is similar to GDB, and instructions for using commands can be viewed with <cmd> help.

To install the Crash Toolkit and kernel debug packets using crash:

Crash

Kernel-debuginfo-common

Kernel-debuginfo

Crash use

Analyze Linux crash dump data or a live system.

Crash [OPTION] namelist memory-image (dumpfile form)

Crash [OPTION] [NameList] (Live system form)

To debug a vmcore using crash, you need at least two parameters:

NameList: Uncompressed kernel image file Vmlinux, default at/usr/lib/debug/lib/modules/$ (Uname-r)/vmlinux, by

The Kernel debugging information pack is provided.

Memory-image: Memory dump file Vmcore, default is/var/crash/%host-%date/vmcore, generated by Kdump.

For example: # crash/usr/lib/debug/lib/modules/$ (uname-r)/vmlinux/var/crash/%host-%date/vmcore

(1) Error type

You can first view the type of error in Vmcore-dmesg.txt, such as:

1. Divide error:0000 [#1] SMP, with a divisor of 0 causing the kernel to crash, triggered by the 1th CPU.

2. bug:unable to handle kernel null pointer dereference at 000000000000012C, reference null pointer.

This allows you to know the type of error that caused the kernel to crash.

(2) Wrong location

RIP for the instructions that caused the kernel to crash, call Trace is the function call stack, and the calling path of the function can be determined by RIP and calls Trace, and the

Which of the functions in which the instruction raised the error.

For example, RIP is: [<ffffffff812cdb54>]? tcp_enter_loss+0x1d3/0x23b

[<ffffffff812cdb54>] is the virtual address of the instruction in memory.

Tcp_enter_loss is the function name (symbol).

0x1d3 is the deviation of this instruction from the Tcp_enter_loss entry, 0x23b is the length after the function is compiled into machine code.

This allows you to determine which function caused the error, and the approximate location of the error.

Call Trace is the invocation stack of functions, which is viewed from the bottom up. Can be used to parse the invocation relationship of a function.

(3) Crash basic output

# crash/usr/lib/debug/lib/modules/$ (uname-r)/vmlinux/var/crash/%host-%date/vmcore

      KERNEL:/usr/lib/debug/lib/modules/2.6.32-358.el6.x86_64/vmlinux
    dumpfile:vmcore  [PARTIAL DUMP]
        CPUS :
        Date:fri Sep 16:47:01 2014 uptime:7 days
      , 06:37:46
LOAD average:0.19, 0.05,
       0.01 tasks:282
  
   nodename:localhost.localdomain
     release:2.6.32-358.el6.x86_64
     VERSION: #1 SMP Tue Oct 10:18:21 CST 2013< C11/>machine:x86_64  (1999 MHZ)
      memory:48 GB
       PANIC: "oops:0002 [#1] SMP" (check log for details)
         PI d:0
     COMMAND: "Swapper"
        task:ffffffff81a8d020  (1 of)  [thread_info:ffffffff81a00000]
         cpu:0
       state:task_running (PANIC)

  

These basic output information is straightforward and can be triggered by the SYS command.

(4) Crash common commands

BT: Print function Call stack, displays a task ' s kernel-stack backtrace, can specify process number BT <pid>.

LOG: Print system message buffer, displays the kernel log_buf contents, such as Log | Tail-n 30.

PS: Displays the status of the process,> represents an active process, such as PS | grep RU.

SYS: Displays the system overview.

Kmem-i: Displays memory usage information.

Dis <addr>: Disassembly of a given address.

Exception RIP is the instruction that caused the error.

About log command:

The kernel first prints the message to the kernel-state ring buffer, and the user-state KLOGD is responsible for reading and forwarding it to the SYSLOGD for recording to disk.

When the kernel crashes, messages may not be logged to disk, but the ring buffer typically has records. So log commands can be viewed sometimes.

To the missing information in the system log.

(5) Structural body and variable

View the values of all members in the structure body, for example:

# PS | grep RU

>     0      0   0  ffffffff81a8d020  RU   0.0       0      0  [Swapper]

# struct Task_struct ffffffff81a8d020

struct Task_struct {state
  = 0, 
  stack = 0xffffffff81a00000, 
  usage = {
    counter = 2
  }, 
  

Shows the definition of the entire structure:

# struct TASK_STRUCT

struct Task_struct {
    volatile long int state;
    void *stack;
    atomic_t usage;
    unsigned int flags;

Shows the definition of the entire structure and the offset of each member:

# Struct-o Task_struct

struct Task_struct {
     [0] volatile long int state;
     [8] void *stack;
    [atomic_t] usage;
    [unsigned] int flags;
    ...

Displays the definition of the member in the structure body, along with its offset:

# struct TASK_STRUCT.PID

struct Task_struct {
  [1192] pid_t pid;
}

To display the values of the members in the structure body:

# struct Task_struct.pid ffffffff81a8d020

  PID = 0

To view the value of a global variable:

# p Sysctl_tcp_rmem

Sysctl_tcp_rmem = $ $ = 
 {40960, 873800, 41943040}

View the PERCPU global variable (prefix per_cpu_):

# p Per_cpu__irq_stat

PER-CPU DATA Type:
  irq_cpustat_t per_cpu__irq_stat//variable type declaration
per-cpu ADDRESSES:
  [0]: ffff880028216540// Number No. 0 CPU corresponding variable address
  [1]: ffff880645416540 ...
  

View the value of the No. 0 CPU corresponding variable:

# struct irq_cpustat_t ffff880028216540

struct irq_cpustat_t {
  __softirq_pending = 0, 
  __nmi_count = 4780195, Irq0_irqs 
  = 148, 
  ...

(6) Disassembly and source line

Disassembly:

# dis ffffffffa021ba91//disassembly of an instruction

# dis-l probe_2093+497 10/Disassembly 10 instructions starting from an address

For symbols in the kernel:

# sym TCP_V4_DO_RCV//through symbol, display virtual address and source location

# Sym ffffffff8149f930//via virtual address, display symbol and source location

For symbols in a module:

Need to load the corresponding module in order to display the symbol corresponding to the source code:

# MoD//View module

# mod-s Module/path/to/module.ko//Loading module

# SYM symbol//display symbols corresponding to the module source code, can also use virtual address

(7) Modify memory

Provides dynamic modification of the runtime kernel functionality for debugging, but is not allowed on Rhel and CentOS.

Wr:modifies the contents of memory.

WR [-u |-k |-P] [-8 | -16 | -32 | -64] [address | symbol] Value

Use examples:

# p Sysctl_tcp_timestamps

Sysctl_tcp_timestamps = $ = 1

# WR Sysctl_tcp_timestamps 0

Wr:cannot Write to/dev/crash!

I'll rub it,/dev/crash. The file attribute is RW, but the crash_fops does not provide a write function, so it is read-only.

This feature is useful, but is prohibited by Rhel and CentOS, so if you want to dynamically modify the running kernel or use the Systemtap bar.

Reference

[1]. http://www.ibm.com/developerworks/cn/linux/l-cn-kdump1/

[2]. http://www.ibm.com/developerworks/cn/linux/l-cn-kdump2/

[3]. http://www.ibm.com/developerworks/cn/linux/l-cn-kdump3/

[4]. http://www.ibm.com/developerworks/cn/linux/l-cn-kdump4/

[5]. http://www.ibm.com/developerworks/cn/linux/l-cn-dumpanalyse/

[6]. http://people.redhat.com/anderson/crash_whitepaper/