Ambari Metrics Collector Mobile Log directory after starting an error

This morning the cluster space is full, mobile Ambari Metrics collector Log Directory after the service can not start, the log is as follows:

Java.net.ConnectException:Connection refused at Sun.nio.ch.SocketChannelImpl.checkConnect (Native Method) At Sun.nio.ch.SocketChannelImpl.finishConnect (Socketchannelimpl.java:712) at Org.apache.zookeeper.ClientCnxnSocketNIO.doTransport (Clientcnxnsocketnio.java:361) at Org.apache.zookeeper.clientcnxn$sendthread.run (Clientcnxn.java:1141)2018- -- -  One: $: at,686INFO org.apache.zookeeper.ClientCnxn:Opening socket connection to server node2.bigdata/192.168.0.129:61181. Won't attempt to authenticate using SASL (unknown error)2018- -- -  One: $: at,686WARN org.apache.zookeeper.ClientCnxn:Session0x0  forServerNULL, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException:Connection refused At Sun.nio.ch.SocketChannelImpl.checkConnect (Native Method) at Sun.nio.ch.SocketChannelImpl.finishConnect (Sock Etchannelimpl.java:712) at Org.apache.zookeeper.ClientCnxnSocketNIO.doTransport (Clientcnxnsocketnio.java:361) at Org.apache.zookeeper.clientcnxn$sendthread.run (Clientcnxn.java:1141)2018- -- -  One: $: at,788WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:Possibly Transient zookeeper, Quorum=node2.bigdata:61181, Exception=org.apache.zookeeper.keeperexception$connectionlossexception:keepererrorcode = ConnectionLoss for/ams-hbase-unsecure/meta-region-Server2018- -- -  One: $: at,788ERROR Org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:ZooKeeper GetData failed after4attempts2018- -- -  One: $: at,788WARN org.apache.hadoop.hbase.zookeeper.zkutil:hconnection-0x72cc7e6f0x0, Quorum=node2.bigdata:61181, Baseznode=/ams-hbase-unsecure unable to get data of znode/ams-hbase-unsecure/meta-region-Serverorg.apache.zookeeper.keeperexception$connectionlossexception:keepererrorcode= Connectionloss for/ams-hbase-unsecure/meta-region-Server at Org.apache.zookeeper.KeeperException.create (Keeperexception.java: About) at Org.apache.zookeeper.KeeperException.create (Keeperexception.java:Wuyi) at Org.apache.zookeeper.ZooKeeper.getData (Zookeeper.java:1212) at Org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData (Recoverablezookeeper.java:354) at Org.apache.hadoop.hbase.zookeeper.ZKUtil.getData (Zkutil.java:624) at Org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState (Metatablelocator.java:491) at Org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation (Metatablelocator.java:172) at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable (Metatablelocator.java:611) at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable (Metatablelocator.java:592) at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable (Metatablelocator.java:565) at Org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation (Zookeeperregistry.java: A) at Org.apache.hadoop.hbase.client.connectionmanager$hconnectionimplementation.locatemeta ( Connectionmanager.java:1209) at Org.apache.hadoop.hbase.client.connectionmanager$hconnectionimplementation.locateregion ( Connectionmanager.java:1176) at Org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations ( Rpcretryingcallerwithreadreplicas.java:340) at Org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call (Scannercallablewithreplicas.java:159) at Org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call (Scannercallablewithreplicas.java: A) at Org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries (Rpcretryingcaller.java:211) at Org.apache.hadoop.hbase.client.ClientScanner.call (Clientscanner.java:327) at Org.apache.hadoop.hbase.client.ClientScanner.nextScanner (Clientscanner.java:302) at Org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction (Clientscanner.java:167) at Org.apache.hadoop.hbase.client.ClientScanner.<init> (Clientscanner.java:162) at Org.apache.hadoop.hbase.client.HTable.getScanner (Htable.java:799) at Org.apache.hadoop.hbase.MetaTableAccessor.fullScan (Metatableaccessor.java:602) at Org.apache.hadoop.hbase.MetaTableAccessor.tableExists (Metatableaccessor.java:366) at Org.apache.hadoop.hbase.client.HBaseAdmin.tableExists (Hbaseadmin.java:415) at org.apache.phoenix.query.connectionqueryservicesimpl$ -. Call (Connectionqueryservicesimpl.java:2363)

Finally find a solution:

79552347

Save the monitoring data in the/var/lib/ambari-metrics-collector/directory of the three folders checkpoint, HBase and hbase-tmp deleted (can be backed up), restart Metrics collector is OK.

rm -rf/var/lib/ambari-metrics-collector/*

Ambari Metrics Collector Mobile Log directory after starting an error