Linux process causes CPU card to die

Source: Internet
Author: User
Tags zookeeper

Reprint Please specify source: http://blog.csdn.net/guoyjoe/article/details/49924557

One, the mailbox received a bunch of monitoring, the alarm inside empty roughly as follows, it is clear that the CPU is not enough, IO also a bit of a problem: Host: bwebser2__10.253.5.198 time: 2015.11.15 15:25:17 Status: Problem level: Warning Alarm reason: Processor load is too high on bwebser2 content: Processor Load (1 min average per core): value=52.53 original event id:30605 host: bwebser2__10.253.5.198 time: 2015.11.18 15:42:23 Status: Problem level: Warning alarm Reason: Disk I/O is overloaded on bwebser2 content: CPU IOW AIT time:value=68.7% original event id:30812

Second, with the top view process, found that there are nearly 2000 processes

[Email protected] ~]# toptop-10:00:32 up 184 days, 19:55,  2 users,  load average:49.39, 52.06, 53.04tasks:1826 Total,   1 running, 1825 sleeping,   0 stopped,   0 zombiecpu (s): 22.5%us,  3.8%sy,  0.0%ni, 31.7%id, 41.3 %wa,  0.7%hi,  0.0%si,  0.0%stmem:   8058056k Total,  7631808k used,   426248k free,   718780k Buffersswap:        0k Total,        0k used,        0k free,   358720k Cached


Third, speculation may be related to SendMail, check maillog log, has been the police: No space left on device

[[email protected] ~]# tail-f/var/log/maillog Nov 10:12:15 bwebser2 postfix/postdrop[19470]: Warning:mail_ Queue_enter:create file Maildrop/878633.19470:no space left on Devicenov 10:12:15 bwebser2 postfix/postdrop[27287]: w Arning:mail_queue_enter:create file Maildrop/900082.27287:no space left on Devicenov 10:12:15 bwebser2 postfix/postd ROP[12347]: warning:mail_queue_enter:create file Maildrop/919377.12347:no space left on Devicenov 10:12:15 Bwebser2 POSTFIX/POSTDROP[21222]: warning:mail_queue_enter:create file Maildrop/937001.21222:no space left on Devicenov 19 10:12 : Bwebser2 postfix/postdrop[25028]: warning:mail_queue_enter:create file Maildrop/956095.25028:no space left on Devic  Enov 10:12:16 bwebser2 postfix/postdrop[28123]: warning:mail_queue_enter:create file Maildrop/980022.28123:no space Left on Devicenov 10:12:16 bwebser2 postfix/postdrop[26680]: Warning:mail_queue_enter:create file maildrop/999360.26 680:no space left on DevIce 


Iv. using lsof to determine the number of sendmail and postdrop processes, the number of processes reached more than 2000, why so many?

[[email protected] ~]# lsof |grep sendmail |wc-l24682[[email protected] ~]# lsof |grep postdrop  |wc-l24108



Five, view the file index node Inode, found that the space is full:

[Email protected] log]# df-ifilesystem       inodes   iused    IFree iuse% mounted on/dev/xvda1      1310720 1310720        0  100%/tmpfs           1007257       1  1007256    1%/dev/shm/dev/xvdb1     13107200    6142 13101058    1%/u01 with df-th command: [email protected] statistics]# df-thfilesystem     Type   Size  used Avail use% mounted ON/DEV/XVDA1     ext4    20G  4.1G   15G  22%/tmpfs          tmpfs  3.9G     0  3.9G   0 %/dev/shm/dev/xvdb1     ext3   197G   18G  170G  10%/u01


Six, clear the Zookeeper monitoring log to free up the root of the space

cd/home/zookeeper/monitor [[email protected] monitor]# lltotal 8drwxrwxr-x 163 zookeeper zookeeper 4096 Nov 00:16 Char Tsdrwxrwxr-x 167 Zookeeper Zookeeper 4096 Nov 17:31 statistics[[email protected] monitor]# CD CHARTSRM-RF *[[email Pro Tected] monitor]# CD. /statistics/[[email protected] statistics]# rm-rf 201506*[[email protected] statistics]# RM-RF 201507*[[email protected ] statistics]# rm-rf 201508*[[email protected] statistics]# rm-rf 201509*[[email protected] statistics]# RM-RF 201510*


Vii. after killing all sendmail and Postdrop processes

[[Email protected] ~] #ps-ef|grep SendMail | Grep-v grep | awk  ' {print ' kill-9 ' $ |sh[[email protected] ~] #ps-ef|grep Postdrop | Grep-v grep | awk  ' {print ' kill-9 ' $ |sh '


Eight, lsof view, the number of processes is 0

[[email protected] ~]# lsof |grep sendmail |wc-l0[[email protected] ~]# lsof |grep postdrop  |wc-l0


Nine, the neglected/etc/cron.d under the Sysstat, modifies the sysstat, the operation is as follows:

[[email protected] CRON.D] #cd/etc/cron.d/[[email protected] cron.d]# lltotal 12-rw-r--r--. 1 root root 113  0hourly-rw-r--r--. 1 root root 108 Apr  7  raid-check-rw-r--r--. 1 root root 23  5 Nov  sysstat vi sysstat Add &>/dev/null# Run System activity Accounting tool every MINUTES*/10 * * * ROOT/USR/LIB/SA/SA1 1 1 &>/dev/null# generate a daily summary of process accounting at 23:5353 * * * ROOT/USR /lib/sa/sa2-a &>/dev/null  


Ten, again with the top command to view the process only more than 100, monitoring the alarm disappears, the problem is done!

[[Email protected] cron.d]# service SendMail restartsendmail:unrecognized service[[email protected] cron.d]# toptop-10: 43:12 up 184 days, 20:37,  2 users,  load average:1.03, 1.54, 14.15tasks:105 Total,   1 running, 104 sleeping,
   0 stopped,   0 zombiecpu (s): 43.4%us,  1.3%sy,  0.0%ni, 47.9%id,  7.0%wa,  0.3%hi,  0.0%si,  0.0%stmem:   8058056k Total,  6762996k used,  1295060k free,  1422060k buffersswap:        0k Total,        0k used,        0k free,   381392k Cached




Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Linux process causes CPU card to die

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.