Zabbix monitoring tomcat troubleshooting with "Connection refused to host: 127.0.0.1"

Zabbix monitoring tomcat troubleshooting with "Connection refused to host: 127.0.0.1"
Zabbix monitors tomcat for "Connection refused to host: 127.0.0.1" fault resolution

Description

A tomcat monitored by jmx suddenly fails to collect data, which is very strange! Because other tomcat services are good. Check zabbix_server.log and find the following error message:

{"response":"failed","error":"java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: \n\tjava.net.ConnectException: Connection refused (Connection refused)"}

Preliminary analysis

The connection to 127.0.0.1 in the log is rejected. In fact, the ip address configured in zabbix is 10.0.x.x, which is not 127.0.0.1 at all! In addition, the ip addresses of other tomcat monitoring configurations on the local machine are the same as those of 10.0.x.x, and data can be obtained normally. Therefore, ip configuration errors are basically ruled out.

Further Analysis

Capture packets to see if any new discoveries exist.

14:15:23.813599 IP 10.0.1.21.52018 > 10.0.1.2.35029: Flags [P.], seq 16:241, ack 72, win 114, options [nop,nop,TS val 1187352905 ecr 1080255901], length 225    0x0000:  4500 0115 b888 4000 4006 6b44 0a00 0115  E.....@.@.kD....    0x0010:  0a00 0102 cb32 88d5 40b7 29d1 cb06 c15b  .....2..@.)....[    0x0020:  8018 0072 1458 0000 0101 080a 46c5 9149  ...r.X......F..I    0x0030:  4063 659d 51ac ed00 0577 0f01 b0de 5ddd  @ce.Q....w....].    0x0040:  0000 015b cc31 2f2b 8cd2 7372 002e 6a61  ...[.1/+..sr..ja    0x0050:  7661 782e 6d61 6e61 6765 6d65 6e74 2e72  vax.management.r    0x0060:  656d 6f74 652e 726d 692e 524d 4953 6572  emote.rmi.RMISer    0x0070:  7665 7249 6d70 6c5f 5374 7562 0000 0000  verImpl_Stub....    0x0080:  0000 0002 0200 0070 7872 001a 6a61 7661  .......pxr..java    0x0090:  2e72 6d69 2e73 6572 7665 722e 5265 6d6f  .rmi.server.Remo    0x00a0:  7465 5374 7562 e9fe dcc9 8be1 651a 0200  teStub......e...    0x00b0:  0070 7872 001c 6a61 7661 2e72 6d69 2e73  .pxr..java.rmi.s    0x00c0:  6572 7665 722e 5265 6d6f 7465 4f62 6a65  erver.RemoteObje    0x00d0:  6374 d361 b491 0c61 331e 0300 0070 7870  ct.a...a3....pxp    0x00e0:  7732 000a 556e 6963 6173 7452 6566 0009  w2..UnicastRef..    0x00f0:  3132 372e 302e 302e 3100 00d7 7d5b 1a9d  127.0.0.1...}[..    0x0100:  530b aa35 bab0 de5d dd00 0001 5bcc 312f  S..5...]....[.1/    0x0110:  2b80 0101 78                             +...x

Expected

The tomcat jmx returned to zabbix-java-gateway contains the "UnicastRef .. 127.0.0.1" in the packet captured.

Compare normal host packet capture

14:14:28.086601 IP 10.0.1.23.52018 > 10.0.1.2.35622: Flags [P.], seq 16:241, ack 72, win 227, options [nop,nop,TS val 1124559044 ecr 1080200175], length 225    0x0000:  4500 0115 3e7b 4000 4006 e54f 0a00 0117  E...>{@.@..O....    0x0010:  0a00 0102 cb32 8b26 66da 56c9 6880 ce04  .....2.&f.V.h...    0x0020:  8018 00e3 1e4e 0000 0101 080a 4307 68c4  .....N......C.h.    0x0030:  4062 8bef 51ac ed00 0577 0f01 4794 88c8  @b..Q....w..G...    0x0040:  0000 015b cccf 59be 4b10 7372 002e 6a61  ...[..Y.K.sr..ja    0x0050:  7661 782e 6d61 6e61 6765 6d65 6e74 2e72  vax.management.r    0x0060:  656d 6f74 652e 726d 692e 524d 4953 6572  emote.rmi.RMISer    0x0070:  7665 7249 6d70 6c5f 5374 7562 0000 0000  verImpl_Stub....    0x0080:  0000 0002 0200 0070 7872 001a 6a61 7661  .......pxr..java    0x0090:  2e72 6d69 2e73 6572 7665 722e 5265 6d6f  .rmi.server.Remo    0x00a0:  7465 5374 7562 e9fe dcc9 8be1 651a 0200  teStub......e...    0x00b0:  0070 7872 001c 6a61 7661 2e72 6d69 2e73  .pxr..java.rmi.s    0x00c0:  6572 7665 722e 5265 6d6f 7465 4f62 6a65  erver.RemoteObje    0x00d0:  6374 d361 b491 0c61 331e 0300 0070 7870  ct.a...a3....pxp    0x00e0:  7732 000a 556e 6963 6173 7452 6566 0009  w2..UnicastRef..    0x00f0:  3130 2e30 2e31 2e32 3300 0085 7ece aadd  10.0.1.23...~...    0x0100:  5f44 ea91 6647 9488 c800 0001 5bcc cf59  _D..fG......[..Y    0x0110:  be80 0101 78                             ....x

Initial Fault Locating

By comparing the data returned by a normal host to the local ip address rather than the 127 ip address, it should be a problem in the setup or code of the faulty tomcat. The reason for network exclusion may be that other tomcat monitoring rooms on the local machine are normal.

Exclude code and Configuration

Compare the fault with the normal tomcat configuration file and the war md5 code to confirm the consistency and eliminate the possibility of configuration and code.

I think the network still needs confirmation

Comparison of network listening for Fault-Tolerant tomcat

Fault tomat monitoring

[root@web01 bin]# lsof -p 20911 |grep LISTENjava    20911 tomcat   16u     IPv4          511652279       0t0       TCP *:55165 (LISTEN)java    20911 tomcat   17u     IPv4          511652280       0t0       TCP *:52018 (LISTEN)java    20911 tomcat   18u     IPv4          511652281       0t0       TCP *:53596 (LISTEN)java    20911 tomcat   37u     IPv4          511652282       0t0       TCP *:51018 (LISTEN)java    20911 tomcat   67u     IPv4          511651274       0t0       TCP localhost:53018 (LISTEN)

Normal tomcat listener

java    24715 tomcat   16u     IPv4          293717104        0t0       TCP *:34174 (LISTEN)java    24715 tomcat   17u     IPv4          293717105        0t0       TCP *:52018 (LISTEN)java    24715 tomcat   18u     IPv4          293717106        0t0       TCP *:36197 (LISTEN)java    24715 tomcat   37u     IPv4          293717107        0t0       TCP *:51018 (LISTEN)java    24715 tomcat   38u     IPv4          293713856        0t0       TCP localhost:53018 (LISTEN)

Comparative Analysis

I found that both tomcat listened to localhost, Which is 127.0.0.1. Is it because some tomcat returns the local ip address and some return the localhost ip address?

To confirm, google uses the keyword "zabbix jmx 127.0.0.1"

The first one is!

ConfigureJMX-Zabbix. orghttps: // www.zabbix.org/wiki/configurejmxApril 19, 2016 How-How Zabbix monitors JMX? Zabbix connects to Java... ConnectException: Connection refused to host: 127.0.0.1; nested exception is: java.net.

It turns out that

Add the parameter Djava. rmi. server. hostname = $ REMOTE_HOST_IP_ADDR when tomcat starts.

The test script in this document is included.

#!/usr/bin/env bashZBXGET="/usr/bin/zabbix_get"if [ $# != 5 ]then    echo "Usage: $0 
  
   
    
     
     
      "    exit;fiQUERY="{\"request\": \"java gateway jmx\",\"conn\": \"$3\",\"port\": $4,\"keys\": [\"$5\"]}"$ZBXGET -s $1 -p $2 -k "$QUERY"
     
    
   
  
 

Troubleshooting Process

Modify the tomcat Startup File and add-Djava. rmi. server. hostname = web01

[root@web01 ~]# vi /etc/init.d/tomcat-xxxxexport CATALINA_OPTS="-Dcom.sun.management.jmxremote \               -Djava.rmi.server.hostname=web01 \               -Dcom.sun.management.jmxremote.port=52018 \               -Dcom.sun.management.jmxremote.ssl=false \               -Dcom.sun.management.jmxremote.authenticate=false"

Restart tomcat

 [root@web01 ~]# systemctl daemon-reload[root@web01 ~]# systemctl restart tomcat-xxxx

Test OK

[root@ops01 ~]# ./zabbix_get_jmx 127.0.0.1 10052 web01 52018 'jmx[\"java.lang:type=Memory\",HeapMemoryUsage.max]'{"data":[{"value":"1013448704"}],"response":"success"}

Packet Capture observation OK

15:13:20.219330 IP 10.0.1.21.52018 > 10.0.1.2.48027: Flags [P.], seq 16:237, ack 72, win 114, options [nop,nop,TS val 1190829298 ecr 1083732308], length 221    0x0000:  4500 0111 acc2 4000 4006 770e 0a00 0115  E.....@.@.w.....    0x0010:  0a00 0102 cb32 bb9b 1e00 305e 18d6 66fb  .....2....0^..f.    0x0020:  8018 0072 0e70 0000 0101 080a 46fa 9cf2  ...r.p......F...    0x0030:  4098 7154 51ac ed00 0577 0f01 e4ad b8e9  @.qTQ....w......    0x0040:  0000 015b d24b 0070 81c7 7372 002e 6a61  ...[.K.p..sr..ja    0x0050:  7661 782e 6d61 6e61 6765 6d65 6e74 2e72  vax.management.r    0x0060:  656d 6f74 652e 726d 692e 524d 4953 6572  emote.rmi.RMISer    0x0070:  7665 7249 6d70 6c5f 5374 7562 0000 0000  verImpl_Stub....    0x0080:  0000 0002 0200 0070 7872 001a 6a61 7661  .......pxr..java    0x0090:  2e72 6d69 2e73 6572 7665 722e 5265 6d6f  .rmi.server.Remo    0x00a0:  7465 5374 7562 e9fe dcc9 8be1 651a 0200  teStub......e...    0x00b0:  0070 7872 001c 6a61 7661 2e72 6d69 2e73  .pxr..java.rmi.s    0x00c0:  6572 7665 722e 5265 6d6f 7465 4f62 6a65  erver.RemoteObje    0x00d0:  6374 d361 b491 0c61 331e 0300 0070 7870  ct.a...a3....pxp    0x00e0:  772e 000a 556e 6963 6173 7452 6566 0005  w...UnicastRef..    0x00f0:  7765 6230 3100 00ea d5a7 39b1 1943 203b  web01.....9..C.;    0x0100:  44e4 adb8 e900 0001 5bd2 4b00 7080 0101  D.......[.K.p...    0x0110:  78                                       x

Acknowledgment and reference

ConfigureJMX