Zookeeper monitoring indicators

Currently, there are two ways to get the Zookeeper monitoring indicator:

1. Get a variety of monitoring metrics with Zookeeper's four-letter words command

2, through the JMX client connection Zookeeer external Mbean to obtain monitoring metrics (need to modify the startup script to enable it to support remote JMX connection)

The indicators obtained in the above two ways are broadly consistent.

Use the following commands to obtain these monitoring information echo commands | NC IP Port

Such as:

echo conf | NC 192.168.144.110 2181

now, the monitoring that can be obtained is broadly listed:

conf:

The ability to obtain configuration information to the zookeeper, including

Client Port,

Data and log paths,

Interval unit time,

Limit the number of connections per server to a single client side,

Time-out,

ServerID, etc. information,

Follower during the boot process, all the latest data is synchronized from leader, and then the starting state of the external service is determined. Leader allows F to complete this work within initlimit time.

During the operation, leader is responsible for communicating with all the machines in the ZK cluster, for example, through some heartbeat detection mechanisms, to detect the machine's survival status. If L make a heartbeat packet after Synclimit and have not received a response from F, then I think this f is out of line.

Election port.

Detailed information is as follows:

clientport=2181
datadir=/export/servers/zookeeper-3.4.6/data/version-2
datalogdir=/export/servers/ Zookeeper-3.4.6/logs/version-2
ticktime=2000
maxclientcnxns=60
minsessiontimeout=4000
maxsessiontimeout=40000
serverid=2
initlimit=10
synclimit=5
electionalg=3
electionport= 3888
quorumport=2888
peertype=0

Cons:

Overview of the connection information,

Connect IP

Port number

The number of packets for this connection

The number of packets received for this connection

Session Id of the connection

Last operation mode/command

Time stamp of the connection

Time-out (not confirmed)

The last Zxid

The last response time stamp

Time delay information for the connection

such as

Detailed information is as follows:

/192.168.147.102:56168[1] (queued=0,recved=60,sent=60,sid=0x24b3673bb141d0e,lop=ping,est=1422614959584,to=40000 , lcxid=0x9,lzxid=0xffffffffffffffff,lresp=1422615627690,llat=1,minlat=0,avglat=0,maxlat=1)/ 192.168.162.16:43107[1] (queued=0,recved=619,sent=643,sid=0x24b3673bb1419fa,lop=ping,est=1422610618969,to=30000 , lcxid=0x7c,lzxid=0xffffffffffffffff,lresp=1422615623122,llat=0,minlat=0,avglat=1,maxlat=296)/ 192.168.162.16:43110[1] (queued=0,recved=411,sent=427,sid=0x14b36741ee41b17,lop=ping,est=1422611808516,to=30000 , lcxid=0x6d,lzxid=0xffffffffffffffff,lresp=1422615622028,llat=0,minlat=0,avglat=3,maxlat=444)/ 192.168.144.107:38698[1] (queued=0,recved=2,sent=2,sid=0x24b3673bb141da3,lop=na,est=1422615628413,to=20000, lcxid=0x1,lzxid=0x68000ad897,lresp=1422615629458,llat=2,minlat=0,avglat=1,maxlat=2)/192.168.162.16:43117[1] ( Queued=0,recved=62,sent=62,sid=0x24b3673bb141d05,lop=ping,est=1422614913239,to=40000,lcxid=0x8,lzxid= 0xffffffffffffffff,lresp=1422615620045,llat=1,minlat=0,avglat=1,maxlat=53)/192.168.144.98:34702[1] (queued=0,recved=5027,sent=5027,sid=0x24b3673bb14001f,lop=ping,est=1422548712199 , to=40000,lcxid=0xc,lzxid=0xffffffffffffffff,lresp=1422615622652,llat=1,minlat=0,avglat=1,maxlat=1151)/ 192.168.144.98:34135[1] (queued=0,recved=7111,sent=7216,sid=0x24b3673bb140007,lop=ping,est=1422548006229,to= 30000,LCXID=0X17D,LZXID=0XFFFFFFFFFFFFFFFF,LRESP=1422615626940,LLAT=1,MINLAT=0,AVGLAT=1,MAXLAT=1004)/ 192.168.162.16:43109[1] (queued=0,recved=779,sent=832,sid=0x4b3673ce4a1a4d,lop=ping,est=1422611214674,to=30000, lcxid=0x7db,lzxid=0xffffffffffffffff,lresp=1422615623541,llat=0,minlat=0,avglat=1,maxlat=468)/ 192.168.147.102:56039[1] (queued=0,recved=382437,sent=382437,sid=0x24b3673bb141c7a,lop=getd,est=1422614382595, to=20000,lcxid=0x5d5e4,lzxid=0x68000ad89b,lresp=1422615629750,llat=0,minlat=0,avglat=0,maxlat=570)/ 192.168.147.101:44124[1] (queued=0,recved=19853228,sent=19853228,sid=0x24b3673bb140000,lop=getd,est= 1422547738803,to=20000,lcxid=0x12eefa9,lzxid=0x68000ad89b,lresp=1422615629750,llat=0,minlat=0,avglat=0,maxlat=8325)/192.168.147.102:39271[1] (queued=0,recved=4819,sent= 4820,sid=0x24b3673bb1400cc,lop=ping,est=1422551694373,to=40000,lcxid=0x19,lzxid=0xffffffffffffffff,lresp= 1422615627807,llat=0,minlat=0,avglat=1,maxlat=893)/192.168.144.107:45476[1] (queued=0,recved=18796922,sent= 18796922,sid=0x24b3673bb14038c,lop=getd,est=1422555547627,to=20000,lcxid=0x11ed178,lzxid=0x68000ad89b,lresp= 1422615629750,llat=0,minlat=0,avglat=0,maxlat=8325)/192.168.162.16:43124[0] (queued=0,recved=1,sent=0)/ 192.168.144.106:47163[1] (queued=0,recved=1919891,sent=1919891,sid=0x24b3673bb14003e,lop=getd,est=1422549512849 , to=20000,lcxid=0x1d4b8f,lzxid=0x68000ad897,lresp=1422615629580,llat=1,minlat=0,avglat=0,maxlat=2678)/ 172.17.36.11:32728[1] (queued=0,recved=6009,sent=6021,sid=0x24b3673bb1403ca,lop=ping,est=1422555925304,to=30000 , lcxid=0x31,lzxid=0xffffffffffffffff,lresp=1422615622868,llat=0,minlat=0,avglat=1,maxlat=513)/ 192.168.144.102:34378[1] (queued=0,recved=6643,SENT=6751,SID=0X24B3673BB1401D2,LOP=PING,EST=1422553000472,TO=30000,LCXID=0X192,LZXID=0XFFFFFFFFFFFFFFFF, lresp=1422615623376,llat=1,minlat=0,avglat=1,maxlat=880)/192.168.162.16:43108[1] (queued=0,recved=379,sent=379, sid=0x24b3673bb141a00,lop=ping,est=1422610718273,to=40000,lcxid=0xa,lzxid=0xffffffffffffffff,lresp= 1422615629070,llat=0,minlat=0,avglat=2,maxlat=291) null[0] (queued=0,recved=3,sent=2,sid=0x4b305d40f933dd,lop= Getc,est=1422615629686,to=30000,lcxid=0x1934a,lzxid=0x68000ad89b,lresp=1422615629689,llat=0,minlat=0,avglat=0, maxlat=0)/192.168.162.16:43112[1] (queued=0,recved=376,sent=386,sid=0x4b3673ce4a1ab7,lop=ping,est=1422612058410 , to=30000,lcxid=0x54,lzxid=0xffffffffffffffff,lresp=1422615621245,llat=0,minlat=0,avglat=1,maxlat=483)/ 192.168.147.102:56038[1] (queued=1,recved=382733,sent=382732,sid=0x24b3673bb141c79,lop=getd,est=1422614380708, to=20000,lcxid=0x5d70b,lzxid=0x68000ad89b,lresp=1422615629748,llat=59,minlat=0,avglat=0,maxlat=570)/ 192.168.144.103:45100[1] (queued=0,recved=6816,sent=6903,sid=0x24b3673bb140040,lop=ping,est=1422549539691,to=30000,lcxid=0xe7,lzxid= 0xffffffffffffffff,lresp=1422615623049,llat=0,minlat=0,avglat=1,maxlat=909)/192.168.144.107:45488[1] (queued=1, Recved=18790392,sent=18790391,sid=0x24b3673bb140391,lop=getd,est=1422555569021,to=20000,lcxid=0x11eb7f5,lzxid= 0x68000ad89b,lresp=1422615629748,llat=56,minlat=0,avglat=0,maxlat=8325)/192.168.144.107:38694[1] (queued=0, recved=743,sent=743,sid=0x24b3673bb141da2,lop=getd,est=1422615626179,to=20000,lcxid=0x2e6,lzxid=0x68000ad89b, lresp=1422615629750,llat=0,minlat=0,avglat=0,maxlat=59)/192.168.162.16:43115[1] (queued=0,recved=159,sent=159, sid=0x24b3673bb141c69,lop=ping,est=1422614308831,to=30000,lcxid=0x1b,lzxid=0xffffffffffffffff,lresp= 1422615624623,llat=0,minlat=0,avglat=2,maxlat=71)/192.168.144.107:38688[1] (queued=0,recved=1442,sent=1442,sid= 0x24b3673bb141da1,lop=getd,est=1422615623769,to=20000,lcxid=0x5a1,lzxid=0x68000ad89b,lresp=1422615629750,llat= 0,minlat=0,avglat=0,maxlat=170) 

Crst:

Reset connection state, is an execute operation is not a select operation

Returns a status message after execution:

Connection stats Reset.

Dump:

Output all messages waiting in the queue for sessions and temporary nodes

0x24b3673bb140000:
     /magpie/workerbeats/11011599
0x14b36741ee41de4:
     /phenix/servers/px0000000816
     /phenix/mygroups/1
0x14b049fe56b89e5:
     /hbase/rs/hhz111,60021,1422454057830
0x4b305d40f92989:
     /hbase/rs/hhz115,60021,1422521527024
0x14b36741ee41edc:
     /magpie/workerbeats/3502573
0X24B3673BB141DC6:
     /magpie/workerbeats/3002570

ENVI:

Environment information for the current server:

Version information

Host of hosts

JVM-related parameters: Version,classpath,lib, etc.

OS-related parameters: Name,version, etc.

Current host user information: Name,dir, etc.

environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT host.name=hhz112 java.version=1.7.0_60 Java.vendor=oracle Corporation Java.home=/export/servers/jdk1.7.0_60/jre java.class.path=/export/servers/ zookeeper-3.4.6/bin/. /build/classes:/export/servers/zookeeper-3.4.6/bin/. /build/lib/*.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/slf4j-log4j12-1.6.1.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/slf4j-api-1.6.1.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/netty-3.7.0.final.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/log4j-1.2.16.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/jline-0.9.94.jar:/export/servers/zookeeper-3.4.6/bin/. /zookeeper-3.4.6.jar:/export/servers/zookeeper-3.4.6/bin/. /src/java/lib/*.jar:/export/servers/zookeeper-3.4.6/bin/. /conf:/export/servers/zookeeper-3.4.6/bin/. /build/classes:/export/servers/zookeeper-3.4.6/bin/. /build/lib/*.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/slf4j-log4j12-1.6.1.jar:/export/servers/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/netty-3.7.0.final.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/log4j-1.2.16.jar:/export/servers/zookeeper-3.4.6/bin/. /lib/jline-0.9.94.jar:/export/servers/zookeeper-3.4.6/bin/. /zookeeper-3.4.6.jar:/export/servers/zookeeper-3.4.6/bin/. /src/java/lib/*.jar:/export/servers/zookeeper-3.4.6/bin/. /conf:.:/ Export/servers/jdk1.6.0_25/lib/dt.jar:/export/servers/jdk1.6.0_25/lib/tools.jar java.library.path=/usr/java/
Packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.io.tmpdir=/tmp java.compiler=<na> Os.name=Linux Os.arch=amd64 os.version=2.6.32-358.el6.x86_64 user.name=hhz user.home=/home/hhz user.dir=/export/servers/ zookeeper-3.4.6

Ruok:

Query the current server state if normal return Imok

Imok

Srst:

The same is an execute operation instead of SELECT, which resets the server state:

Server stats Reset.

SRVR:

Brief information about the server:

Version

Delay

Number of bags received

Number of contracts

Number of connections

Status and other information

Zookeeper version:3.4.6-1569965, built on 02/20/2014 09:09 GMT
Latency min/avg/max:0/0/182
received:97182
sent:97153
connections:22
outstanding:8
zxid:0x68000af381
mode:follower
Node count:101065

Stat:

Some state and connection information is a combination of the preceding information:

Zookeeper version:3.4.6-1569965, built on 02/20/2014 09:09 GMT clients:/192.168.147.102:56168[1] (queued=0,recved=41, sent=41)/192.168.144.102:34378[1] (queued=0,recved=54,sent=54)/192.168.162.16:43108[1] (queued=0,recved=40,sent= /192.168.144.107:39948[1] (queued=0,recved=1421,sent=1421)/192.168.162.16:43112[1] (queued=0,recved=54,sent= /192.168.162.16:43107[1] (queued=0,recved=54,sent=54)/192.168.162.16:43110[1] (queued=0,recved=53,sent=53)/ 192.168.144.98:34702[1] (queued=0,recved=41,sent=41)/192.168.144.98:34135[1] (queued=0,recved=61,sent=65)/ 192.168.162.16:43109[1] (queued=0,recved=54,sent=54)/192.168.147.102:56038[1] (queued=0,recved=165313,sent= 165314)/192.168.147.102:56039[1] (queued=0,recved=165526,sent=165527)/192.168.147.101:44124[1] (queued=0,recved= 162811,sent=162812)/192.168.147.102:39271[1] (queued=0,recved=41,sent=41)/192.168.144.107:45476[1] (queued=0, recved=166422,sent=166423)/192.168.144.103:45100[1] (queued=0,recved=54,sent=54)/192.168.162.16:43133[0] (queued=0,recved=1,sent=0)/192.168.144.107:39945[1] (queued=0,recved=1825,sent=1825)/ 192.168.144.107:39919[1] (queued=0,recved=325,sent=325)/192.168.144.106:47163[1] (queued=0,recved=17891,sent= 17891)/192.168.144.107:45488[1] (queued=0,recved=166554,sent=166555)/172.17.36.11:32728[1] (queued=0,recved=54, sent=54)/192.168.162.16:43115[1] (queued=0,recved=54,sent=54) Latency min/avg/max:0/0/599 received:224869 Sent: 224817 connections:23 outstanding:0 zxid:0x68000af707 mode:follower Node count:101081

Wchs:

Number of connections with watch path and the number of path and watcher of Watch

Connections watching 102 paths total
watches:172

Wchc

Connect all paths for monitoring: (Consider cons command information integration)

0x24b3673bb14001f
	/hbase/root-region-server
	/hbase/master

WCHP:

path is monitored by those connections: (consider consolidating the cons command information)

/dubbo/feedinterface/configurators
	0x4b3673ce4a1a4d
/dubbo/userinterface/providers
	0x14b36741ee41b17
	0x4b3673ce4a1a4d
	0x24b3673bb1401d2
	0x4b3673ce4a1ab7

mntr:

Various metrics for monitoring the health status of the zookeeper server:

Version

Delay

Receiving package

Employer

Number of connections

Number of client requests not completed

Leader/follower status

Number of Znode

Watch Count

Temporary node points

Approximate data size should be a sum of values

Number of open file descriptors

Maximum number of file descriptors

Number of Fllower

Wait a minute

Zk_version	3.4.6-1569965, built on 02/20/2014 09:09 GMT
zk_avg_latency	0
zk_max_latency	2155
zk_min_latency	0
zk_packets_received	64610660
zk_packets_sent	64577070
zk_num_ Alive_connections
zk_outstanding_requests	0
zk_server_state	leader
Zk_znode_ Count	101125
zk_watch_count	315
zk_ephemerals_count	633
zk_approximate_data_size	27753592
zk_open_file_descriptor_count
zk_max_file_descriptor_count	4096
Zk_ Followers	2
zk_synced_followers	2
zk_pending_syncs	0

These are all information indicators that are based on the 3.4 version of Zookeeper four letter words.