How to fix hung_task_timeout_secs and blocked for more than seconds problem

Author:skate
time:2015/03/04

How to fix Hung_task_timeout_secs and blocked for more than seconds problem

Phenomenon: System Hang live, can ping, but SSH is not responding

View Message Log
[1379100.801689] [<ffffffff81536f95>] page_fault+0x25/0x30
[1379100.801693] Info:task java:710923 blocked for more than seconds.
[1379100.801766] not tainted 2.6.32-042stab104.1 #1
[1379100.801835] "echo 0 >/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1379100.801963] Java D ffff8800372d7200 0 710923 709954 67084186 0x00000000
[1379100.801968] ffff880e57e71cf0 0000000000000082 ffffea00021a8fc0 ffff880e57e71c68
[1379100.801972] ffffffff81155c60 ffff8800372d7200 ffffea00021a8fc0 ffff88100c409638
[1379100.801976] 00000007FA23BFFC ffff880e57e71c78 ffffffff81155cd1 ffff880e57e71ca8
[1379100.801980] Call Trace:
[1379100.801984] [<ffffffff81155c60>]? __lru_cache_add+0x40/0x90
[1379100.801988] [<ffffffff81155cd1>]? lru_cache_add_lru+0x21/0x40
[1379100.801992] [<ffffffff81172c9c>]? handle_pte_fault+0x65c/0x1040
[1379100.801996] [<ffffffff81536705>] Rwsem_down_failed_common+0x95/0x1d0
[1379100.802000] [<ffffffff81536896>] rwsem_down_read_failed+0x26/0x30
[1379100.802004] [<ffffffff812a6a34>] call_rwsem_down_read_failed+0x14/0x30
[1379100.802008] [<ffffffff81535d94>]? down_read+0x24/0x30
[1379100.802011] [<ffffffff8104dffe>] __do_page_fault+0x18e/0x480
[1379100.802015] [<ffffffff8106f0c8>]? finish_task_switch+0xc8/0x120
[1379100.802019] [<ffffffff81539c2e>] do_page_fault+0x3e/0xa0
[1379100.802022] [<ffffffff81536f95>] page_fault+0x25/0x30
show  Vitaly Medvedev added a comment-yesterday 10:34 PM [1379100.801682] [<ffffffff81015019>]? read_tsc+0x9/0x20 [1379100.801685] [<ffffffff81539c2e>] do_page_fault+0x3e/0xa0 [1379100.801689] [< FFFFFFFF81536F95&GT] page_fault+0x25/0x30 [1379100.801693] Info:task java:710923 blocked for more than seconds. [1379100.801766] Not tainted 2.6.32-042stab104.1 #1 [1379100.801835] "echo 0 >/proc/sys/kernel/hung_task_timeout_ secs "Disables this message. [1379100.801963] Java D ffff8800372d7200 0 710923 709954 67084186 0x00000000 [1379100.801968] ffff880e57e71cf0 0000000000 000082 ffffea00021a8fc0 ffff880e57e71c68 [1379100.801972] Ffffffff81155c60 ffff8800372d7200 ffff88100c409638 [1379100.801976] 00000007FA23BFFC ffff880e57e71c78 ffffffff81155cd1 ffff880e57e71ca8 [ 1379100.801980] Call Trace: [1379100.801984] [<ffffffff81155c60>]? __lru_cache_add+0x40/0x90 [1379100.801988] [<ffffffff81155cd1>]? lru_cache_add_lru+0x21/0x40 [1379100.801992] [<ffffffff81172c9c>]? handle_pte_fault+0x65c/0x1040 [1379100.801996] [<ffffffff81536705>] rwsem_down_failed_common+0x95/0x1d0 [ 1379100.802000] [<ffffffff81536896>] rwsem_down_read_failed+0x26/0x30 [1379100.802004] [<ffffffff812a6a34 ] call_rwsem_down_read_failed+0x14/0x30 [1379100.802008] [<ffffffff81535d94>]? down_read+0x24/0x30 [1379100.802011] [<ffffffff8104dffe>] __do_page_fault+0x18e/0x480 [1379100.802015] [< Ffffffff8106f0c8>]? finish_task_switch+0xc8/0x120 [1379100.802019] [<ffffffff81539c2e>] do_page_fault+0x3e/0xa0 [1379100.802022] [<ffffffff81536f95>] page_fault+0x25/0x30

Host's load reaches about 460

By default Linux uses up to 40% of the available memory for file system caching.
After this mark has been reached the "file system flushes all outstanding" data to
Disk causing all following IOs going synchronous. For flushing out this data to
Disk This there was a time limit to seconds by default. Here's the case.
IO subsystem isn't fast enough to flush the data withing seconds. As IO
subsystem responds slowly and more requests are served, System Memory gets
Up resulting in the above error, thus serving HTTP requests.

Solution:

1. Modifying parameters Vm.dirty_ratio and Vm.dirty_backgroud_ratio can avoid this problem

# sysctl-w vm.dirty_ratio=10
# sysctl-w Vm.dirty_background_ratio=5

Immediate effect:
# sysctl-p

Permanent modification (requires reboot):
# vi/etc/sysctl.conf
Vm.dirty_background_ratio = 5
Vm.dirty_ratio = 10

2. Find the process of good resources and then optimize it

Reference: http://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/

-------End-------