Record A zabbix troubleshooting Process

Zabbix has been used for a while, and has encountered many major and minor errors. I personally have a habit of occasionally tail-f the log of zabbix_server, which may be the relationship between obsessive-compulsive disorder and do not like to see any error message in the log.

10.1 was supposed to have a holiday, but the day before he came home, he suddenly found that the following error was frequently reported in the log.

zabbix_server [4615]: ERROR [file:json.c,line:714] Something impossible has just happened.zabbix_server [4615]: ERROR [file:json.c,line:714] Something impossible has just happened.

Although I have not found any impact on the overall operation of zabbix for the time being, the frequent errors in the log made me very difficult. At that time, I had to catch a train to go home for a holiday, so I was not responsible for the time being, when I got home, I always remembered this thing, bored at night.

By the way, connect to the server for analysis. First, according to the error, Google found nothing big. Okay, let's take a look at the 714 lines of source code in JSON. C. What is this.

# locate json.c/workspace/download/redis-2.8.7/deps/lua/src/lua_cjson.c/workspace/zabbix-2.0.11/src/libs/zbxjson/json.c

The following is the code near this line. I can understand Nima. The C language is weak. I can't help it. I had to enable the most awesome debug function of zabbix.

 713                     default: 714                         THIS_SHOULD_NEVER_HAPPEN; 715                 }

Modify the configuration file, restart zabbix-server, and the log will soar instantly. Then, filter the logs and wait until an error is reported.

#tail -f zabbix_server.log | grep "Something impossible has just happened."

After an error is reported, I manually pull the log to the local machine to see what the problem is.

  4615:20141002:002356.188 In lld_check_record() jp_row:‘{"{#API_WEB}\}":"getSecretKey"}‘  4684:20141002:002356.188 query [txnlev:0] [select min(clock) from history where itemid=24155]zabbix_server [4615]: ERROR [file:json.c,line:714] Something impossible has just happened.

  4559:20141002:002447.962 In substitute_discovery_macros() data:‘{#SERVICES_METHODS}‘zabbix_server [4559]: ERROR [file:json.c,line:714] Something impossible has just happened.

According to the preceding log analysis, something impossible has just happened is reported every time the obtained variables are automatically discovered.

This error shows the two monitoring programs that automatically monitor some API interfaces added to a development a few days ago. I manually stopped the monitoring of both APIs.

No error is reported. I carefully checked the above and found a suspicious location. There is an "\}" behind api_web in the brackets below "\}"

{"{#API_WEB}\}":"getSecretKey"}‘

Basically, the problem has been located. In the script of the automatic discovery interface, an "\}" is added, leading to the above error.

Log on to the server and modify the script. No error is reported in the log .. After two days, I finally managed to solve the problem. After the solution is completed, I am in a good mood, smoke a cigarette, feel comfortable, and then I am broken down ~! = ^ _ ^ =

Record A zabbix troubleshooting Process