Softlock-kernel panic-not syncing: softlockup: hung tasks

The first time a softlockup bug was solved, the Fae reply information could not properly analyze the root cause of the problem, and the previous understanding of softlockup was superficial,

So how can we make the kernel generate a softlockup warning.

 

Basic Principles:

Each CPU starts a kernel thread, which uses the current time to update the global time file of a percpu, and checks the current time in the tick interrupt.

Is the difference between this and the global time file greater than a set threshold, for the linux-2.6.27, this threshold is set to 60 s, if the 60 s kernel thread has not

If you have a chance to get a schedule, it means there is a deadlock on the CPU.

 

Before clock_was_set in do_settimefoday, add

Preempt_disable ();

While (1); this status is IRQ enabled, but the preemption is disabled, and no task can seize the settimeofday task on cpadministration.

The softlockup kernel thread on the CPU is not running. If the difference between the global time file and the current time exceeds the threshold, this situation is detected during tick interruption.

 

Next, run a small program that calls settimefoday. After 60 s, the following warning occurs:

 

 

 

 

Try it on arm in the same way

BUG: Soft lockup-CPU #2 stuck for 22 s! [Settimeofday: 1, 485]
Modules linked in:

PID: 485, COMM: settimeofday
CPU: 2 not tainted (3.3.0-RC3 + #48)
PC is at do_settimeofday + 0xd8/0xe4
LR is at do_settimeofday + 0xd7/0xe4
PC: [<800491e8>] LR: [<800491e7>] PSR: 40000033Arm-non-linux-addr2line-e vmlinuxWhich code line can be obtained for 800491e8?
SP: 86d97f30 IP: 00000000 FP: 00000000
R10: 00000000 R9: 86d96000 R8: 806dc144
R7: 00000001 R6: 00000421 R5: 806dc140 R4: 80711180
R3: 806e2d9c R2: 00000000 r1: 00000422 R0: 00000000
Flags: nzcv irqs on fiqs on mode svc_32 ISA thumb segment user
Control: 50c53c7d table: 66dd806a DAC: 00000015
[<80012639>] (unwind_backtrace + 0x1/0xa4) from [<8005ddf9>] (watchdog_timer_fn + 0x145/0x190)
[<8005ddf9>] (watchdog_timer_fn + 0x145/0x190) from [<80036f91>] (hrtimer_run_queues + 0xf9/0x240)
[<80036f91>] (hrtimer_run_queues + 0xf9/0x240) from [<80027a01>] (run_local_timers + 0xd/0x18)
[<80027a01>] (run_local_timers + 0xd/0x18) from [<80027a33>] (update_process_times + 0x27/0x54)
[<80027a33>] (update_process_times + 0x27/0x54) from [<8004d4a3>] (tick_periodic + 0x4b/0xb0)
[<8004d4a3>] (tick_periodic + 0x4b/0xb0) from [<8004d593>] (tick_handle_periodic + 0x23/0x78)
[<8004d593>] (tick_handle_periodic + 0x23/0x78) from [<80012153>] (twd_handler + 0x23/0 x2c)
[<80012153>] (twd_handler + 0x23/0 x2c) from [<80060a5b>] (handle_percpu_devid_irq + 0x73/0 xc8)
[<80060a5b>] (handle_percpu_devid_irq + 0x73/0 xc8) from [<8005e163>] (generic_handle_irq + 0x23/0x34)
[<8005e163>] (generic_handle_irq + 0x23/0x34) from [<8000df8b>] (handle_irq + 0x4f/0x94)
[<8000df8b>] (handle_irq + 0x4f/0x94) from [<80008431>] (gic_handle_irq + 0x29/0x84)
[<80008431>] (gic_handle_irq + 0x29/0x84) from [<8000d1bf>] (_ irq_svc + 0x3f/0x80)
Exception stack (0x86d97ee8 to 0x86d97f30)
7ee0: 00000000 00000422 00000000 806e2d9c 80711180 806dc140
7f00: 00000421 00000001 806dc144 86d96000 00000000 00000000 00000000 86d97f30
7f20: 800491e7 800491e8 40000033 ffffff
[<8000d1bf>](_ Irq_svc + 0x3f/0x80) from [<800491e8>] (do_settimeofday + 0xd8/0xe4) deadlock location in do_settimeofday + 0xd8
[<800491e8>] (do_settimeofday + 0xd8/0xe4) from [<80020ced>] (do_sys_settimeofday + 0x49/0 xA8)
[<80020ced>] (do_sys_settimeofday + 0x49/0 xA8) from [<80020d71>] (sys_settimeofday + 0x25/0 xb8)
[<80020d71>] (sys_settimeofday + 0x25/0 xb8) from [<8000d681>] (ret_fast_syscall + 0x1/0x5c)

 

I tried it and waited for Fae to reply.

 

========================================================== ============================================

333473

BUG: Soft lockup-CPU #0 stuck for 61 s! [Events/0: 51]
Modules linked in: oslxcontext ippathmt KMD kprn x_tables ip6_tables ip_tables IPv6 binfmt_misc 8021q
CPU 0
$0: 0000000000000000 ffffffff81410000 0000000000000003
$4: 0000000000000010 0000000000000010 0000000000000010 a80000041fa0bdd0
$8: 0000000000000000 a80000041fbed080 0000000000000000 0000000000000001
$12: a80000041fa0bfe0 0000000000008c00 ffffff813b4000 00000000009a
$16: ffffffff813d0000 a8000004132faf00 000000000000000f 0000000000000001
$20: 0000000000000000 ffffffff8119a920 0000000000000000 fffffffffffffe
$24: 0000000000000002 ffffffff81103e40
$28: a80000041fa08000 a80000041fa0be00 ffffffff8136e3f0 ffffff811898cc
Hi: 0000000000000000
Lo: 0000000000000000
EPC: ffffffff81189784 smp_call_function_mask + 0x144/0x258
Not tainted
RA: ffffffff811898cc smp_call_function + 0x34/0x78
Status: 10008e3 kx sx ux kernel exl IE
Cause: 40808000
PRID: 000d030b (cavium octeon)
BUG: Soft lockup-CPU #10 stuck for 61 s! <G id = "1"> [ntpdate: 3032]
Kernel panic-not syncing: softlockup: hung tasks
------------ [Cut here] ------------
Warning: At kernel/SMP. C: 332 smp_call_function_mask + 0x250/0x258 ()
Modules linked in: oslxcontext ippathmt KMD kprn x_tables ip6_tables ip_tables IPv6 binfmt_misc 8021q
Call trace:
[<Ffffff813221f4>] dump_stack + 0x8/0x34
<Ffffffff8115f588>] warn_on_slowpath + 0x58/0x90
[<Ffffffff81189890>] smp_call_function_mask + 0x250/0x258
[<Ffffff811898cc>] smp_call_function + 0x34/0x78
[<Ffffff81322428>] Panic + 0x98/0x220
[<Ffffffff81192c10>] softlockup_tick + 0x290/0x310
[<Ffffff8116ce84>] update_process_times + 0x34/0x78
[<Ffffff811852e8>] tick_sched_timer + 0x58/0 xd8
[<Ffffff8117e398>] _ run_hrtimer + 0xb0/0xc8
[<Ffffff8117f150>] hrtimer_interrupt + 0x158/0x210
[<Ffffff81138ac4>] c0_compare_interrupt + 0x11c/0x148
[<Ffffff8119334c>] _ handle_irq_event + 0x5c/0xb8
[<Ffffff81193430>] handle_irq_event + 0x88/0x140
[<Ffffff811950e8>] handle_percpu_irq + 0x78/0 xd8
[<Ffffff81100f28>] plat_irq_dispatch + 0xc8/0x1d0
[<Ffffff81130a00>] ret_from_irq + 0x0/0x4
[<Ffffffff81189784>] smp_call_function_mask + 0x144/0x258
[<Ffffff811898cc>] smp_call_function + 0x34/0x78
[<Ffffff80000652c>] on_each_cpu + 0x2c/0x88
[<Ffffff81175410>] run_workqueue + 0xc8/0x1a0
[<Ffffff81175960>] worker_thread + 0x60/0 xd0
[<Ffffffff8117a404>] kthread + 0x5c/0xb0
[<Ffffff81133328>] kernel_thread_helper + 0x10/0x18
 
--- [End trace 1770b5013095d915] ---
Rebooting in 10 seconds... <3>
Modules linked in: oslxcontext ippathmt KMD kprn x_tables ip6_tables ip_tables IPv6 binfmt_misc 8021q
CPU 10
$0: 0000000000000000 a80000041cf3be70 0000000000000003 0000000000000000
$4: 0000000000000010 0000000000000010 0000000000000010 a80000041cf3bca0
$8: 0000000000000000 1000000003b9aca00 ffffffff81410000 ffffffff81412520
$12: 0000000000000000 ffffffffc0000008 ffffffff81191258 fffffffff0000000
$16: ffffffff813d0000 a80000041feaca80 000000000000000f 0000000000000001
$20: 0000000000000000 ffffffff8117e610 0000000000000000
$24: 0000000000000042 ffffffff81103e40
$28: a80000041cf38000 a80000041cf3bcd0 10810013fe0 ffffff811898cc
Hi: 03d0900000000000
Lo: 010000ab1_000000
EPC: ffffffff81189778 smp_call_function_mask + 0x138/0x258
Tainted: G W
RA: ffffffff811898cc smp_call_function + 0x34/0x78
Status: 10008e3 kx sx ux kernel exl IE
Cause: 40808000
PRID: 000d030b (cavium octeon)
Kernel panic-not syncing: softlockup: hung tasks
------------ [Cut here] ------------
Warning: At kernel/SMP. C: 332 smp_call_function_mask + 0x250/0x258 ()
Modules linked in: oslxcontext ippathmt KMD kprn x_tables ip6_tables ip_tables IPv6 binfmt_misc 8021q
Call trace:
[<Ffffff813221f4>] dump_stack + 0x8/0x34
<Ffffffff8115f588>] warn_on_slowpath + 0x58/0x90
[<Ffffffff81189890>] smp_call_function_mask + 0x250/0x258
[<Ffffff811898cc>] smp_call_function + 0x34/0x78
[<Ffffff81322428>] Panic + 0x98/0x220
[<Ffffffff81192c10>] softlockup_tick + 0x290/0x310
[<Ffffff8116ce84>] update_process_times + 0x34/0x78
[<Ffffff811852e8>] tick_sched_timer + 0x58/0 xd8
[<Ffffff8117e398>] _ run_hrtimer + 0xb0/0xc8
[<Ffffff8117f150>] hrtimer_interrupt + 0x158/0x210
[<Ffffff81138ac4>] c0_compare_interrupt + 0x11c/0x148
[<Ffffff8119334c>] _ handle_irq_event + 0x5c/0xb8
[<Ffffff81193430>] handle_irq_event + 0x88/0x140
[<Ffffff811950e8>] handle_percpu_irq + 0x78/0 xd8
[<Ffffff81100f28>] plat_irq_dispatch + 0xc8/0x1d0
[<Ffffff81130a00>] ret_from_irq + 0x0/0x4
[<Ffffffff81189778>] smp_call_function_mask + 0x138/0x258
[<Ffffff811898cc>] smp_call_function + 0x34/0x78
[<Ffffff80000652c>] on_each_cpu + 0x2c/0x88
[<Ffffffff81181ad4>] do_settimeofday + 0x14c/0x180
[<Ffffff81178e0c>] sys_clock_settime + 0xec/0x108
[<Ffffffff811912d0>] compat_sys_clock_settime + 0x78/0x90
[<Ffffff81141b64>] handle_sysn32 + 0x44/0x84