The method of locating kernel module crash

In the kernel module development process, the system crashes often occur, at this time the system freezes, unable to locate and analyze problems.

Common positioning method is the installation of Kdump-tools,kdump-tools can be frozen before the kernel log saved, so that after the boot can analyze the last crash log.

The installation configuration method of Kdump-tools is not introduced here, how to analyze the crash log, find out the wrong place in the code.

Kdump-tools crash Log is generally placed in/var/crash/error time/DMESG. The time directory, such as/var/crash/201706131703/dmesg.201706131703, opens this file, as follows:

[1493201.293587] buflen=2097152,gwid=223344,addr=33554671
[1493258.160173] fq=300 Full,will is change FQ
[1493258.160179] Max_gw_buf_len0=81984,max_gw_buf_len1=0
[1493258.160199] bug:unable to handle kernel NULL pointer dereference at 0000000000000028
[1493258.160204] IP: [<ffffffffc02ef10a>] search_fq_to_insert+0x1d2/0x239 [Hnrcore]
[1493258.160216] PGD 0
[1493258.160219] oops:0000 [#1] SMP
[1493258.160222] Modules linked In:binfmt_misc fou (OE) hnrcore (OE) iptable_filter Xt_nat iptable_nat nf_conntrack_ipv4 N F_defrag_ipv4 Nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipip tunnel4 ip_tunnel ip6_udp_tunnel udp_tunnel Bonding Joydev input_leds intel_powerclamp coretemp KVM ipmi_ssif ipmi_devintf irqbypass Gpio_ich Crct10dif_pclmul Crc32_pclmul 8 250_fintek Dcdbas shpchp aesni_intel serio_raw aes_x86_64 lrw gf128mul glue_helper lpc_ich ablk_helper I7core_edac cryptd Edac_core ipmi_si ipmi_msghandler acpi_power_meter mac_hid PARPORT_PC ppdev LP Parport autofs4 hid_generic psmouse Usbhid HID PATA_ACPI Megaraid_sas bnx2 fjes [last Unloaded:fou]
[1493258.160271] cpu:0 pid:0 comm:swapper/0 tainted:g IOE 4.4.0-21-generic #37-ubuntu
[1493258.160274] Hardware Name:dell Inc. PowerEdge r710/0xdx06, BIOS 2.2.10 11/09/2010
[1493258.160276] task:ffffffff81e11500 ti:ffffffff81e00000 task.ti:ffffffff81e00000
[1493258.160278] rip:0010:[<ffffffffc02ef10a>] [<ffffffffc02ef10a>] search_fq_to_insert+0x1d2/0x239 [ Hnrcore]
[1493258.160286] Rsp:0018:ffff88032f603ca8 eflags:00010097
[1493258.160288] rax:0000000000000000 rbx:ffff88032f60dd00 rcx:ffff8801a6101000
[1493258.160290] rdx:0000000000000001 rsi:0000000000000000 rdi:ffff88032f603ce0
[1493258.160292] Rbp:ffff88032f603ce8 r08:000000000000000a r09:ffff88032f603cd0
[1493258.160294] r10:0000000000000020 R11:0000000000000BFB r12:ffffffffc02f37c0
[1493258.160296] r13:0000000000000100 r14:ffffffffc02e9bdb r15:0000000000000000
[1493258.160299] fs:0000000000000000 (0000) gs:ffff88032f600000 (0000) knlgs:0000000000000000
[1493258.160301] cs:0010 ds:0000 es:0000 cr0:000000008005003b
[1493258.160303] cr2:0000000000000028 cr3:0000000001e0a000 cr4:00000000000006f0
[1493258.160304] Stack:
[1493258.160306] ffff88032f603d18 ffff8801a6101000 0000000100000000 0000000000000001
[1493258.160310] 0000000000000000 0000000000000000 0000000000000000 fa723263f77ef18e
[1493258.160313] ffff88032f603d48 ffffffffc02ef5bb ffffffffc02f3b30 0000000200000001
[1493258.160316] Call Trace:
[1493258.160318] <IRQ>
[1493258.160325] [<ffffffffc02ef5bb>] frequency_buf_full_process+0x35f/0x567 [HNRcore]
[1493258.160331] [<ffffffffc02ef82d>] frequency_change_main+0x39/0x50 [Hnrcore]
[1493258.160337] [<ffffffffc02ee19f>] alloc_data_per_timer+0x6b0/0xbd9 [Hnrcore]
[1493258.160343] [<ffffffffc02eed69>] fq_alloc_timer+0xa9/0x278 [Hnrcore]
[1493258.160348] [<ffffffffc02e9217>]? gw_manage_via_hfc+0x49/0x49 [Hnrcore]
[1493258.160356] [<ffffffff810fe4f0>]? tick_sched_handle.isra.14+0x60/0x60
[1493258.160361] [<ffffffffc02e9c08>] gw_send_ts_timer+0x2d/0x5e [Hnrcore]
[1493258.160368] [<ffffffff810ec345>] call_timer_fn+0x35/0x120
[1493258.160373] [<ffffffffc02e9bdb>]? GW_SEND_TS_PROCESS+0X9C4/0X9C4 [Hnrcore]
[1493258.160377] [<ffffffff810eccfa>] Run_timer_softirq+0x23a/0x2f0
[1493258.160383] [<ffffffff810859a1>] __do_softirq+0x101/0x290
[1493258.160387] [<ffffffff81085ca3>] Irq_exit+0xa3/0xb0
[1493258.160393] [<ffffffff81826fa2>] smp_apic_timer_interrupt+0x42/0x50
[1493258.160398] [<ffffffff81825262>] apic_timer_interrupt+0x82/0x90
[1493258.160399] <EOI>
[1493258.160405] [<ffffffff816bb9ee>]? Cpuidle_enter_state+0x10e/0x2b0
[1493258.160408] [<ffffffff816bb9df>]? Cpuidle_enter_state+0xff/0x2b0
[1493258.160412] [<ffffffff816bbbc7>] cpuidle_enter+0x17/0x20
[1493258.160418] [<ffffffff810c3d52>] call_cpuidle+0x32/0x60
[1493258.160421] [<ffffffff816bbba3>]? cpuidle_select+0x13/0x20
[1493258.160424] [<ffffffff810c4010>] cpu_startup_entry+0x290/0x350
[1493258.160430] [<ffffffff81817f2c>] rest_init+0x7c/0x80
[1493258.160438] [<ffffffff81f5a011>] Start_kernel+0x481/0x4a2
[1493258.160442] [<ffffffff81f59120>]? early_idt_handler_array+0x120/0x120
[1493258.160445] [<ffffffff81f59339>] x86_64_start_reservations+0x2a/0x2c
[1493258.160448] [<ffffffff81f59485>] x86_64_start_kernel+0x14a/0x16d
[1493258.160450] code:05 3b 2f C0 8b 3b D0 0f 8f b1 FE FF FF C7 D0-EB-8b-D0-MB 8b 44 C5 E8 E0 8b E0 <8b> 3d F9 7f 8b 3d 95 00 00 (in) (E0)
[1493258.160483] RIP [<ffffffffc02ef10a>] search_fq_to_insert+0x1d2/0x239 [Hnrcore]
[1493258.160489] RSP <ffff88032f603ca8>
[1493258.160491] cr2:0000000000000028

How to analyze this crash log, first of all, you can see unable to handle kernel null pointer print, indicating access to the null pointer error, and then look at the location of rip in search_fq_to_insert+0x1d2/0x239 [ Hnrcore], to indicate an error in the execution here code, how to restore the number of lines of this code, you can use the GdB method, see below:

root@cjtx-poweredge-r710:/var/crash/201706131703# Gdb/usr/local/hnr_target/bin/hnrcore.ko
GNU gdb (Ubuntu 7.11-0ubuntu1) 7.11
Copyright (C) 2016 free Software Foundation, Inc.
License gplv3+: GNU GPL version 3 or later This is the free software:you are and redistribute it.
There is NO WARRANTY and to the extent permitted by. Type "Show copying"
and "Show warranty" for details.
This is GDB was configured as "X86_64-linux-gnu".
Type ' show configuration ' for configuration details.
For bugs reporting instructions, please:
Find the GDB manual and other documentation resources online at:
For help, type ' help '.
Type ' apropos word ' to ' search for commands related to Word ...
Reading symbols From/usr/local/hnr_target/bin/hnrcore.ko...done.
(GDB) B *search_fq_to_insert+0x1d2
Breakpoint 1 at 0X9133:FILE/HOME/WORK/HNR/CORE/FREQUENCY_INFO.C, line 762.

You can see that the error appears in line No. 762 of FREQUENCY_INFO.C, as shown in the code:

for (i=0;i<2;i++)
{
FQ = New_fq_tmp[i];
if (Fq->fq_full_count >= 250)///More than 2 seconds of buffer full, this frequency point cannot insert
Continue
if (Fq->gw_num >= MAX_GW_PER_FQ)
Continue

The actual error will be more than the original line, 763 lines, visible here FQ pointer at the end of the decision is empty and directly used, the reason for this.