How to analyze Kernel panic

From: feino Network (www.firnow.com): http://dev.firnow.com/course/6_system/linux/Linuxjs/20091016/179018.html

The below is a example of kernel panic log.

[2, 875.497504] unable to handle kernel paging request at virtual address 4042a00c

[1, 875.499110] PGD = c0004000

[875.501797] [4042a00c] * PGD = 00000000

[2, 875.505364] internal error: Oops: 5 [#1] preempt

[875.509957] modules linked in: DHD

[1, 875.513350] CPU: 0 not tainted (2.6.29 #21)

[875.517877] PC is at lock_acquire + 0x60/0x74

[875.522024] LR is at 0x0

[875.524544] PC: [<c0090e10>] LR: [<00000000>] PSR: 60000093

[875.524557] SP: c042bea0 IP: c042a000 FP: c042becc

[875.536004] R10: 0000e0cd R9: 00000000 R8: 16027ae8

[875.541210] R7: 0000036b R6: 00000000 R5: 4042a000 R4: 60000093

[875.547717] R3: 00000000 R2: 00000000 r1: 00000000 R0: 00000000

[1, 875.554230] flags: nzcv irqs off fiqs on mode svc_32 ISA arm segment Kernel

[875.561610] control: 101_87d table: 1c0ec019 DAC: 00000017

[1, 875.567337] process Swapper (PID: 0, stack Limit = 0xc042a2e0)

[875.573157] Stack: (0xc042bea0 to 0xc042c000)

[875.577490] bea0: 00000002 00000080 00000000 c00735e4 c0036c9c c04d4b40 c00735e4 1d34ce80

[875.585737] bec0: c042bef4 c042bed0 c02fcef8 c0090dbc 00000002 00000000 c00735e4 c04d6010

[875.593984] bee0: rje0cd c04d4b40 c042bf3c c042bef8 c00735e4 c02fcebc c0082d34 c006d870

[875.602230] bf00: c042bf60 00000001 c002e518 c045e1a8 c042bf3c 109e0cd c04d49a0 1d34ce80

[875.610477] bf20: 0000036b 16027ae8 00000000 running 001f c042bf94 c042bf40 c008b1f8 c00735cc

[875.618724] bf40: 00000000 c999a0a0 1d9e6627 0000036b rje0cd c00622a4 60000013 c042f318

[875.626970] bf60: 1d9e6627 0000036b 0000001f c002e518 c042a000 c002e518 c045e1a8 16027ae8

[875.635217] bf80: 510f00f2 2017001f c042bfb4 c042bf98 c002e418 c008b028 c0889a90 c045dd64

[875.643464] bfa0: c0029ea4 c042ec08 c042bfc4 c042bfb8 c02f76a4 c002e3f0 c042bff4 c042bfc8

[875.651710] bfc0: c0008bc0 c02f7644 c0008514 00000000 00000000 c0029ea4 00000000 10c5387d

[875.659957] bfe0: c045e25c c002a2a8 00000000 c042bff8 16008034 c000894c 00000000 00000000

[2, 875.668204] backtrace:

[875.670637] [<c0090db0>] (lock_acquire + 0x0/0x74) from [<c02fcef8>] (_ spin_lock + 0x48/0x58)

[875.678797] R6: 1d34ce80 R5: c00735e4 R4: c04d4b40

[875.683397] [<c02fceb0>] (_ spin_lock + 0x0/0x58) from [<c00735e4>] (get_next_timer_interrupt + 0x24/0x224)

[875.692684] R5: c04d4b40 R4: rje0cd

[875.696244] [<c00735c0>] (get_next_timer_interrupt + 0x0/0x224) from [<c008b1f8>] (tick_nohz_stop_sched_tick + 0x1dc/0x434)

[875.707004] [<c008b01c>] (tick_nohz_stop_sched_tick + 0x0/0x434) from [<c002e418>] (cpu_idle + 0x34/0 x6c)

[875.716204] [<c002e3e4>] (cpu_idle + 0x0/0x6c) from [<c02f76a4>] (rest_init + 0x6c/0x80)

[875.723937] R7: c042ec08 R6: c0029ea4 R5: c045dd64 R4: c0889a90

[875.729577] [<c02f7638>] (rest_init + 0x0/0x80) from [<c0008bc0>] (start_kernel + 0x280/0 x2e4)

[875.737824] [<c0008940>] (start_kernel + 0x0/0x2e4) from [<16008034>] (0x16008034)

[1, 875.745197] Code: e58dc008 e59bc00c e58dc00c ebfffdc6 (e595300c)

[2, 875.751446] Kernel panic-not syncing: Fatal exception

1.

The below is the analysis result:

[2, 875.497504] unable to handle kernel paging request at virtual address 4042a00c

The whole description of the panic.

2.

Modules linked in: DHD

It just means there is one module whose name is DHD. Ko while panic happens.

Not means the panic is in this module.

3.

Process Swapper (PID: 0, stack Limit = 0xc042a2e0)

PID of the panic process is 0.

4.

[875.517877] PC is at lock_acquire + 0x60/0x74

[875.522024] LR is at 0x0

[875.524544] PC: [<c0090e10>] LR: [<00000000>] PSR: 60000093

[875.524557] SP: c042bea0 IP: c042a000 FP: c042becc

[875.536004] R10: 0000e0cd R9: 00000000 R8: 16027ae8

[875.541210] R7: 0000036b R6: 00000000 R5: 4042a000 R4: 60000093

[875.547717] R3: 00000000 R2: 00000000 r1: 00000000 R0: 0000000

The log shows the value of arm registers.

0x74 means the whole length of lock_acquire () function in disassemble format.

0x60 means the key point while panic happens.

GDB the output file which include lock_acquire () function and then disassemble the function to make sure which line has issue.

For example.

Arm-Linux-GDB lockdep. o

Disassemble lock_acquire

5.

[875.573157] Stack: (0xc042bea0 to 0xc042c000)

[875.577490] bea0: 00000002 00000080 00000000 c00735e4 c0036c9c c04d4b40 c00735e4 1d34ce80

[875.585737] bec0: c042bef4 c042bed0 c02fcef8 c0090dbc 00000002 00000000 c00735e4 c04d6010

[875.593984] bee0: rje0cd c04d4b40 c042bf3c c042bef8 c00735e4 c02fcebc c0082d34 c006d870

[875.602230] bf00: c042bf60 00000001 c002e518 c045e1a8 c042bf3c 109e0cd c04d49a0 1d34ce80

[875.610477] bf20: 0000036b 16027ae8 00000000 running 001f c042bf94 c042bf40 c008b1f8 c00735cc

[875.618724] bf40: 00000000 c999a0a0 1d9e6627 0000036b rje0cd c00622a4 60000013 c042f318

[875.626970] bf60: 1d9e6627 0000036b 0000001f c002e518 c042a000 c002e518 c045e1a8 16027ae8

[875.635217] bf80: 510f00f2 2017001f c042bfb4 c042bf98 c002e418 c008b028 c0889a90 c045dd64

[875.643464] bfa0: c0029ea4 c042ec08 c042bfc4 c042bfb8 c02f76a4 c002e3f0 c042bff4 c042bfc8

[875.651710] bfc0: c0008bc0 c02f7644 c0008514 00000000 00000000 c0029ea4 00000000 10c5387d

[875.659957] bfe0: c045e25c c002a2a8 00000000 c042bff8 16008034 c000894c 00000000 00000000

It contains the context of the stack from 0xc042bea0 to 0xc042c000.

6.

[2, 875.668204] backtrace:

[875.670637] [<c0090db0>] (lock_acquire + 0x0/0x74) from [<c02fcef8>] (_ spin_lock + 0x48/0x58)

[875.678797] R6: 1d34ce80 R5: c00735e4 R4: c04d4b40

[875.683397] [<c02fceb0>] (_ spin_lock + 0x0/0x58) from [<c00735e4>] (get_next_timer_interrupt + 0x24/0x224)

[875.692684] R5: c04d4b40 R4: rje0cd

[875.696244] [<c00735c0>] (get_next_timer_interrupt + 0x0/0x224) from [<c008b1f8>] (tick_nohz_stop_sched_tick + 0x1dc/0x434)

[875.707004] [<c008b01c>] (tick_nohz_stop_sched_tick + 0x0/0x434) from [<c002e418>] (cpu_idle + 0x34/0 x6c)

[875.716204] [<c002e3e4>] (cpu_idle + 0x0/0x6c) from [<c02f76a4>] (rest_init + 0x6c/0x80)

[875.723937] R7: c042ec08 R6: c0029ea4 R5: c045dd64 R4: c0889a90

[875.729577] [<c02f7638>] (rest_init + 0x0/0x80) from [<c0008bc0>] (start_kernel + 0x280/0 x2e4)

[875.737824] [<c0008940>] (start_kernel + 0x0/0x2e4) from [<16008034>] (0x16008034)

It means function call stack. From the above log, the call stack is:

Start_kernel ()-> rest_init ()-> cpu_idle ()-> wait ()-> get_next_timer_interrupt ()-> _ spin_lock ()-> lock_acquire ().

7.

[1, 875.745197] Code: e58dc008 e59bc00c e58dc00c ebfffdc6 (e595300c)

It means the latest instruction.

E58dc008 str ip, [Sp, #8]

E59bc00c ldr ip, [FP, #12]

E58dc00c str ip, [Sp, #12]

Ebfffdc6 ......