Ambari Metrics Collector移動日誌目錄後啟動報錯

標籤：locate   r.java   exception   cte   loss   解決辦法   啟動報錯   解決   lib   

今天早上叢集空間滿了，移動Ambari Metrics Collector日誌目錄後服務啟動不了，日誌如下：

java.net.ConnectException: Connection refused        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)2018-07-30 11:46:23,686 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server node2.bigdata/192.168.0.129:61181. Will not attempt to authenticate using SASL (unknown error)2018-07-30 11:46:23,686 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Connection refused        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)2018-07-30 11:46:23,788 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=node2.bigdata:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/meta-region-server2018-07-30 11:46:23,788 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper getData failed after 4 attempts2018-07-30 11:46:23,788 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: hconnection-0x72cc7e6f0x0, quorum=node2.bigdata:61181, baseZNode=/ams-hbase-unsecure Unable to get data of znode /ams-hbase-unsecure/meta-region-serverorg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/meta-region-server        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:624)        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:491)        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:172)        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:611)        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:592)        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:565)        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1209)        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1176)        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:340)        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:159)        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:211)        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:799)        at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)        at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:415)        at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2363)

最後找到解決辦法：

79552347

把儲存監控資料的/var/lib/ambari-metrics-collector/目錄下的三個檔案夾checkpoint 、hbase和hbase-tmp刪了（可以先備份），重啟Metrics Collector就ok了。

rm -rf /var/lib/ambari-metrics-collector/*

 

Ambari Metrics Collector移動日誌目錄後啟動報錯