Linux Kernel 2.6.28 or more bugs, the system runs the first 208.5 days down machine

Brief introduction:

Business Server has a server unexpectedly down machine, the server ping does not pass, unable to log in, I want to through the company KVM System login System Restart resolution, login KVM found the system screen print a large number of kernel errors, KVM cannot use, unable to send restart server. Decisive contact room personnel to help manual restart, manually restart the server after the server running state returned to normal.

Before the server restarts through the KVM management system can see the system screen printing kernel errors, at this time a page screenshot, and then according to the screen printing error keywords, search and trace found that our Linux server kernel bug caused by the problem.

This kernel bug problem will directly cause the server down, find related data found that the kernel bug there are related trigger conditions, the following personal collection of some information to summarize, for everyone to learn from.

1. Official Description:

Sched_clock () overflow after 208.5 days in Linux Kernel

Linux Kernel 2.6. Version 28 above the kernel has this problem.

When the server fails, the system prints errors on the screen roughly as follows:

June 8 14:26:06 Acme Kernel:irq 19:nobody cared (try booting with the "irqpoll" option)

June 8 14:26:06 Acme Kernel:pid:0, Comm:swapper not tainted 2.6.32-220.7.1.el6.x86_64 #1

June 8 14:26:06 Acme Kernel:call Trace:

June 8 14:26:06 Acme kernel: <IRQ> [<ffffffff810db42b>]? __report_bad_irq+0x2b/0xa0

June 8 14:26:06 Acme kernel: [<ffffffff810db62c>]? Note_interrupt+0x18c/0x1d0

June 8 14:26:06 Acme kernel: [<ffffffff810dbd4d>]? Handle_fasteoi_irq+0xcd/0xf0

June 8 14:26:06 Acme kernel: [<ffffffff8100df09>]? Handle_irq+0x49/0xa0

June 8 14:26:06 Acme kernel: [<ffffffff814f4dbc>]? Do_irq+0x6c/0xf0

June 8 14:26:06 Acme kernel: [<ffffffff8100ba53>]? Ret_from_intr+0x0/0x11

June 8 14:26:06 Acme kernel: [<ffffffff8100ba53>]? Ret_from_intr+0x0/0x11

June 8 14:26:06 Acme kernel: <EOI> [<ffffffff812c4b0e>]? intel_idle+0xde/0x170

June 8 14:26:06 Acme kernel: [<ffffffff812c4af1>]? intel_idle+0xc1/0x170

June 8 14:26:06 Acme kernel: [<ffffffff813fa027>]? cpuidle_idle_call+0xa7/0x140

June 8 14:26:06 Acme kernel: [<ffffffff81009e06>]? cpu_idle+0xb6/0x110

June 8 14:26:06 Acme kernel: [<ffffffff814e5ffc>]? start_secondary+0x202/0x245

June 8 14:26:06 Acme Kernel:handlers:

June 8 14:26:06 Acme kernel: [<ffffffffa01e5020>] (rtl8169_interrupt+0x0/0x380 [r8169])

2. After looking for data to analyze this kernel bug by looking for data has 3 trigger conditions;

A:linux System kernel version Linux Kernel 2.6. Version 28 or above (letter 2.8);

B. The system runs for about 208 days;

C. The server uses Pentium 4 or more Intel CPUs (including CPU, 32bit/64bit);

Very sad reminder! Our servers are fully satisfied with the above conditions.

3.bug Related Advisory

The release was released in 2011 and should not have been repaired without a patch.

Kernel-2.6.32-220.4.2.el6 CentOS Fixed This bug in this kernel version

This flaw has been fixed in Linux Kernel 2.6.32.50/3.0.13/3.1.5.

3. Kernel Upgrade

After the troubleshooting found that we have a lot of servers to meet the above kernel bug trigger conditions, and decisively arrange the server plan, to avoid the server unexpectedly down machine things happen again, the following is a simple record of related operations, for your reference:

3.1: Confirm the system version and kernel version

[Email protected] ~]# uname-a

Linux slave2 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT x86_64 x86_64 x86_64 gnu/linux

[Email protected] ~]# more/etc/redhat-release

CentOS Release 6.2 (Final)

[Email protected] ~]# more/proc/version

Linux version 2.6.32-220.el6.x86_64 ([email protected]) (GCC version 4.4.6 20110731 (Red Hat 4.4.6-3) (gcc)) #1 SMP Tue D EC 6 19:48:22 GMT 2011

[Email protected] ~]#

3.2: Kernel Upgrade:

I use the overwrite processing method, execute the following command;

Yum Update-y

After the upgrade is complete, restart the server for the new kernel version to take effect.

Init 6

3.3: Check the upgrade results:

[Email protected] ~]# uname-a

Linux Master 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed June 03:34:52 UTC x86_64 x86_64 x86_64 gnu/linux

[Email protected] ~]# more/etc/redhat-release

CentOS Release 6.4 (Final)

[Email protected] ~]# more/proc/version

Linux version 2.6.32-358.11.1.el6.x86_64 ([email protected]) (GCC version 4.4.7 20120313 (Red Hat 4.4.7-3) (gcc)) #1 SMP Wed June 03:34:52 UTC 2013

After the system reboot is complete, the kernel version has been upgraded to 2.6.32-358, the system is upgraded to 6.4 the latest version.

Redhat official news Release:

Https://rhn.redhat.com/errata/RHBA-2012-0124.html

CentOS official news Release

https://www.centos.org/modules/newbb/viewtopic.php?topic_id=42031

Related information:

https://www.centos.org/modules/newbb/viewtopic.php?topic_id=42031

http://bugs.centos.org/view.php?id=4586#c14466

http://www.ubuntu-tw.org/modules/newbb/viewtopic.php?topic_id=46348

http://bugs.centos.org/view.php?id=5397

Linux Kernel 2.6.28 or more bugs, the system runs the first 208.5 days down machine