Break diagram first network error, wait for seconds

A few days ago, just discussed this problem with a group of friends, he is the SNMP switch when the diagram is intermittent.

I also have this case, but when monitoring Tomcat, Jstat consumes more than 90 CPU,

Cause the normal monitoring project to break the diagram.

In fact, it let us wait for the seconds, then the meaning constantly superimposed 15s, is not 5 minutes after not out of the picture,

Or is the load high not out of the picture?

Case

Zabbix_server.log

12961:20170512:113910.624 Zabbix Agent Item "Java.discovery_status[service_tomcat,all]" on host "172.x.x.x" failed: First network error, wait for seconds

Investigation

Top

Top-14:34:25 up 344 days, 21:11, 1 user, load average: 1.23, 1.09, 1.10

tasks:176 Total, 1 running, 174 sleeping, 1 stopped, 0 zombie

Cpu (s): 25.0%us, 5.0%sy, 0.0%ni, 69.7%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st

mem:3925304k Total, 3672532k used, 252772k free, 174944k buffers

swap:4128764k Total, 79512k used, 4049252k free, 2122448k cached

PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND

2484 Tomcat 0 3333m 30m 12m S 99.8 0.8 9065:51 Jstat

Describes the Jstat exception, which is the Zabbix script used to obtain Tomcat data.

[[email protected] local]# ps -ef|grep jstatroot       2483     1  0 may06 ?         00:00:00 sudo -u tomcat /usr/java/jdk/bin/jstat -gcutil 12470tomcat     2484  2483 99 May06 ?         6-07:03:49 /usr/java/jdk/bin/jstat -gcutil 12470root     17095  32501  0 14:32 pts/0    00:00:00 grep jstat[[email  protected] local]# ps -ef|grep jstatroot      2483      1  0 May06 ?         00:00:00 sudo -u tomcat /usr/java/jdk/bin/jstat -gcutil 12470tomcat     2484  2483 99 may06 ?        6-07:04:20 /usr/java/jdk/bin /jstat -gcutil 12470root     17097 32501  0 14:32  Pts/0    00:00:00 grep jstat solve kill -9 2484 is normal, restart Zabbix_agentd, Observe a load of about 0.5. At first the suspect was a trigger problem, and the trigger was disabled. As to the cause of this problem, it is possible that there are a few of the Python scripts that are forced to kill the Tomcat process jstat_status.py call Jstat, is it a zombie process? Or the cause of this program's deadlock? Is there anything special about Python? 1 more needs to be done. You need to analyze zabbix server-side logs with python or elk to find a lot of not suitable for value  type [numeric  (unsigned)] and data type [decimal]2. A process that occupies cpu 80 above is a problem, This also needs to be added to the monitoring

The other reasons for the broken diagram are as follows:

1. Database optimization such as threads_created | 49258532.zabbix Server Optimization

650) this.width=650; "Src=" https://s2.51cto.com/wyfs02/M01/95/75/wKioL1kVW9nBxPXGAAAVErieoi0037.png-wh_500x0-wm_ 3-wmp_4-s_200992409.png "style=" Float:none; "title=" QQ picture 20170512145250.png "alt=" Wkiol1kvw9nbxpxgaaaverieoi0037.png-wh_50 "/>

650) this.width=650; "Src=" https://s2.51cto.com/wyfs02/M02/95/75/wKiom1kVW9mh7FMMAAARJCSRwQ4311.png-wh_500x0-wm_ 3-wmp_4-s_2311014138.png "style=" Float:none; "title=" QQ picture 20170512145244.png "alt=" Wkiom1kvw9mh7fmmaaarjcsrwq4311.png-wh_50 "/>

3. Refer to the menu of light and light blog, from SQL to troubleshoot HTTP://CAIGUANGGUANG.BLOG.51CTO.COM/1652935/1377089/4. Nic Problem http://www.fyluo.com/?post=1645. ZABBIX Agent executes key, obtains data time exceeds 30s reference http://blog.chinaunix.net/ Uid-8108137-id-4591738.html Solution The maximum time to modify the agent execution key, the source code limits the maximum time of 30S. To modify the source code. and tested only to modify the maximum execution time of the AGETN is not possible, the server or proxy side of the maximum execution time is also 30S. Only modify the agent side 30s,item can capture the mode of active.

The stability of the system is the precondition of the server stability.

This article is from the "Python Operation" blog, declined reprint!

Break diagram first network error, wait for seconds