Linux process causes CPU card to die

Reprint Please specify source: http://blog.csdn.net/guoyjoe/article/details/49924557

One, the mailbox received a bunch of monitoring, the alarm inside empty roughly as follows, it is clear that the CPU is not enough, IO also a bit of a problem: Host: bwebser2__10.253.5.198 time: 2015.11.15 15:25:17 Status: Problem level: Warning Alarm reason: Processor load is too high on bwebser2 content: Processor Load (1 min average per core): value=52.53 original event id:30605 host: bwebser2__10.253.5.198 time: 2015.11.18 15:42:23 Status: Problem level: Warning alarm Reason: Disk I/O is overloaded on bwebser2 content: CPU IOW AIT time:value=68.7% original event id:30812

Second, with the top view process, found that there are nearly 2000 processes

[Email protected] ~]# toptop-10:00:32 up 184 days, 19:55,  2 users,  load average:49.39, 52.06, 53.04tasks:1826 Total,   1 running, 1825 sleeping,   0 stopped,   0 zombiecpu (s): 22.5%us,  3.8%sy,  0.0%ni, 31.7%id, 41.3 %wa,  0.7%hi,  0.0%si,  0.0%stmem:   8058056k Total,  7631808k used,   426248k free,   718780k Buffersswap:        0k Total,        0k used,        0k free,   358720k Cached

Third, speculation may be related to SendMail, check maillog log, has been the police: No space left on device

[[email protected] ~]# tail-f/var/log/maillog Nov 10:12:15 bwebser2 postfix/postdrop[19470]: Warning:mail_ Queue_enter:create file Maildrop/878633.19470:no space left on Devicenov 10:12:15 bwebser2 postfix/postdrop[27287]: w Arning:mail_queue_enter:create file Maildrop/900082.27287:no space left on Devicenov 10:12:15 bwebser2 postfix/postd ROP[12347]: warning:mail_queue_enter:create file Maildrop/919377.12347:no space left on Devicenov 10:12:15 Bwebser2 POSTFIX/POSTDROP[21222]: warning:mail_queue_enter:create file Maildrop/937001.21222:no space left on Devicenov 19 10:12 : Bwebser2 postfix/postdrop[25028]: warning:mail_queue_enter:create file Maildrop/956095.25028:no space left on Devic  Enov 10:12:16 bwebser2 postfix/postdrop[28123]: warning:mail_queue_enter:create file Maildrop/980022.28123:no space Left on Devicenov 10:12:16 bwebser2 postfix/postdrop[26680]: Warning:mail_queue_enter:create file maildrop/999360.26 680:no space left on DevIce 

Iv. using lsof to determine the number of sendmail and postdrop processes, the number of processes reached more than 2000, why so many?

[[email protected] ~]# lsof |grep sendmail |wc-l24682[[email protected] ~]# lsof |grep postdrop  |wc-l24108

Five, view the file index node Inode, found that the space is full:

[Email protected] log]# df-ifilesystem       inodes   iused    IFree iuse% mounted on/dev/xvda1      1310720 1310720        0  100%/tmpfs           1007257       1  1007256    1%/dev/shm/dev/xvdb1     13107200    6142 13101058    1%/u01 with df-th command: [email protected] statistics]# df-thfilesystem     Type   Size  used Avail use% mounted ON/DEV/XVDA1     ext4    20G  4.1G   15G  22%/tmpfs          tmpfs  3.9G     0  3.9G   0 %/dev/shm/dev/xvdb1     ext3   197G   18G  170G  10%/u01

Six, clear the Zookeeper monitoring log to free up the root of the space

cd/home/zookeeper/monitor [[email protected] monitor]# lltotal 8drwxrwxr-x 163 zookeeper zookeeper 4096 Nov 00:16 Char Tsdrwxrwxr-x 167 Zookeeper Zookeeper 4096 Nov 17:31 statistics[[email protected] monitor]# CD CHARTSRM-RF *[[email Pro Tected] monitor]# CD. /statistics/[[email protected] statistics]# rm-rf 201506*[[email protected] statistics]# RM-RF 201507*[[email protected ] statistics]# rm-rf 201508*[[email protected] statistics]# rm-rf 201509*[[email protected] statistics]# RM-RF 201510*

Vii. after killing all sendmail and Postdrop processes

[[Email protected] ~] #ps-ef|grep SendMail | Grep-v grep | awk  ' {print ' kill-9 ' $ |sh[[email protected] ~] #ps-ef|grep Postdrop | Grep-v grep | awk  ' {print ' kill-9 ' $ |sh '

Eight, lsof view, the number of processes is 0

[[email protected] ~]# lsof |grep sendmail |wc-l0[[email protected] ~]# lsof |grep postdrop  |wc-l0

Nine, the neglected/etc/cron.d under the Sysstat, modifies the sysstat, the operation is as follows:

[[email protected] CRON.D] #cd/etc/cron.d/[[email protected] cron.d]# lltotal 12-rw-r--r--. 1 root root 113  0hourly-rw-r--r--. 1 root root 108 Apr  7  raid-check-rw-r--r--. 1 root root 23  5 Nov  sysstat vi sysstat Add &>/dev/null# Run System activity Accounting tool every MINUTES*/10 * * * ROOT/USR/LIB/SA/SA1 1 1 &>/dev/null# generate a daily summary of process accounting at 23:5353 * * * ROOT/USR /lib/sa/sa2-a &>/dev/null  

Ten, again with the top command to view the process only more than 100, monitoring the alarm disappears, the problem is done!

[[Email protected] cron.d]# service SendMail restartsendmail:unrecognized service[[email protected] cron.d]# toptop-10: 43:12 up 184 days, 20:37,  2 users,  load average:1.03, 1.54, 14.15tasks:105 Total,   1 running, 104 sleeping,
   0 stopped,   0 zombiecpu (s): 43.4%us,  1.3%sy,  0.0%ni, 47.9%id,  7.0%wa,  0.3%hi,  0.0%si,  0.0%stmem:   8058056k Total,  6762996k used,  1295060k free,  1422060k buffersswap:        0k Total,        0k used,        0k free,   381392k Cached

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Linux process causes CPU card to die