Reasons for the inability to start after HBase reboot in Unix and solutions

We used several machines to build a cluster and configured Hadoop and HBase according to Hadoop/hbase's "Getting Started" installation. After that, you can start Hadoop and hbase normally, and create a table and insert data.

However, when we visited Master's page: http://10.37.17.252:60010/master.jsp, we found the first problem: in the Regionserver area, There were 2 127.0.0.1 Regionserver, but we didn't set master to Regionserver in Conf/regionservers:

code is as follows

copy code

Region Servers Address Start Code Load 127.0.0.1:60030 1263383321075 requests=0, regions=0, Usedheap=0, Maxheap=0 12 7.0.0.1:60030 1263383321096 requests=0, regions=0, Usedheap=0, maxheap=0 wamdm1.ruc.edu.cn:60030 1263383350174 Requests=0, Regions=0, usedheap=24, maxheap=991 wamdm2.ruc.edu.cn:60030 1263383320980 requests=0, Regions=1, usedheap=32, maxheap=991 wamdm3.ruc.edu.cn:60030 1263383320985 requests=0, Regions=1, usedHeap=33, maxHeap=991 wamdm4.ruc.edu.cn:60030 1263383322246 requests=0, regions=0, usedheap=33, maxheap=991 ...

However, despite the above strange phenomenon, but hbase still seems to be able to work properly. Just when we were going to reboot the hbase, we found the second problem: we tried to run bin/stop-hbase.sh, and then we ran the script that started HBase: bin/ Start-hbase.sh, this time, when we visited the master page, http://10.37.17.252:60010/master.jsp, the following error occurred:

The code is as follows

Copy Code

HTTP error:500 Trying to contacts region server null for region, row ', but failed after 3 attempts. Exceptions: org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1 requesturi=/master.jsp Caused by: Org.apache.hadoop.hbase.client.RetriesExhaustedException:Trying to contacts Region Server null for region, row ", but FAI LED after 3 attempts. Exceptions: org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1 org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1 At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.getregionserverwithretries ( HCONNECTIONMANAGER.JAVA:1002) At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:55) At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:28) At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.listtables (hconnectionmanager.java:433) At Org.apache.hadoop.hbase.client.HBaseAdmin.listTables (hbaseadmin.java:127) At Org.apache.hadoop.hbase.generated.master.master_jsp._jspservice (master_jsp.java:125) At Org.apache.jasper.runtime.HttpJspBase.service (httpjspbase.java:97) At Javax.servlet.http.HttpServlet.service (httpservlet.java:820) At Org.mortbay.jetty.servlet.ServletHolder.handle (servletholder.java:502) At Org.mortbay.jetty.servlet.ServletHandler.handle (servlethandler.java:363) At Org.mortbay.jetty.security.SecurityHandler.handle (securityhandler.java:216) At Org.mortbay.jetty.servlet.SessionHandler.handle (sessionhandler.java:181) At Org.mortbay.jetty.handler.ContextHandler.handle (contexthandler.java:766) At Org.mortbay.jetty.webapp.WebAppContext.handle (webappcontext.java:417) At Org.mortbay.jetty.handler.ContextHandlerCollection.handle (contexthandlercollection.java:230) At Org.mortbay.jetty.handler.HandlerWrapper.handle (handlerwrapper.java:152) At Org.mortbay.jetty.Server.handle (server.java:324) At Org.mortbay.jetty.HttpConnection.handleRequest (httpconnection.java:534) At Org.mortbay.jetty.httpconnection$requesthandler.headercomplete (httpconnection.java:864) At Org.mortbay.jetty.HttpParser.parseNext (httpparser.java:533) At Org.mortbay.jetty.HttpParser.parseAvailable (httpparser.java:207) At Org.mortbay.jetty.HttpConnection.handle (httpconnection.java:403) At Org.mortbay.io.nio.SelectChannelEndPoint.run (selectchannelendpoint.java:409) At Org.mortbay.thread.queuedthreadpool$poolthread.run (queuedthreadpool.java:522) Powered by jetty://

At the same time, in Master's log, the following error occurred:


2010-01-13 18:34:04,424 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:04,425 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:04,425 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:06,425 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:06,429 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:06,430 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:06,430 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:08,430 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:08,434 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:08,435 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:08,435 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:locateregioninmeta Attempt 0 of 3 failed; Retrying after sleep of 2000


org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1


At Org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy (hbaserpc.java:424)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 865)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 881)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locaterootregion (HConnectionManager.java:936)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:581)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:557)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregioninmeta (HConnectionManager.java : 631)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:590)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:557)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.getregionlocation (HConnectionManager.java:407 )


At Org.apache.hadoop.hbase.client.ServerCallable.instantiateServer (servercallable.java:57)


At Org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer (scannercallable.java:62)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.getregionserverwithretries ( hconnectionmanager.java:988)


At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:55)


At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:28)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.listtables (hconnectionmanager.java:433)


At Org.apache.hadoop.hbase.client.HBaseAdmin.listTables (hbaseadmin.java:127)


At Org.apache.hadoop.hbase.generated.master.master_jsp._jspservice (master_jsp.java:125)


At Org.apache.jasper.runtime.HttpJspBase.service (httpjspbase.java:97)


At Javax.servlet.http.HttpServlet.service (httpservlet.java:820)


At Org.mortbay.jetty.servlet.ServletHolder.handle (servletholder.java:502)


At Org.mortbay.jetty.servlet.ServletHandler.handle (servlethandler.java:363)


At Org.mortbay.jetty.security.SecurityHandler.handle (securityhandler.java:216)


At Org.mortbay.jetty.servlet.SessionHandler.handle (sessionhandler.java:181)


At Org.mortbay.jetty.handler.ContextHandler.handle (contexthandler.java:766)


At Org.mortbay.jetty.webapp.WebAppContext.handle (webappcontext.java:417)


At Org.mortbay.jetty.handler.ContextHandlerCollection.handle (contexthandlercollection.java:230)


At Org.mortbay.jetty.handler.HandlerWrapper.handle (handlerwrapper.java:152)


At Org.mortbay.jetty.Server.handle (server.java:324)


At Org.mortbay.jetty.HttpConnection.handleRequest (httpconnection.java:534)


At Org.mortbay.jetty.httpconnection$requesthandler.headercomplete (httpconnection.java:864)


At Org.mortbay.jetty.HttpParser.parseNext (httpparser.java:533)


At Org.mortbay.jetty.HttpParser.parseAvailable (httpparser.java:207)


At Org.mortbay.jetty.HttpConnection.handle (httpconnection.java:403)


At Org.mortbay.io.nio.SelectChannelEndPoint.run (selectchannelendpoint.java:409)


At Org.mortbay.thread.queuedthreadpool$poolthread.run (queuedthreadpool.java:522)


2010-01-13 18:34:08,451 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:08,451 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:08,451 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:10,451 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:10,456 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:10,465 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:10,465 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:12,465 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:12,469 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:12,470 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:14,474 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:14,475 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:14,475 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:16,475 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:16,480 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:16,480 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:16,480 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:18,480 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:18,485 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:18,486 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:18,486 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:locateregioninmeta Attempt 0 of 3 failed; Retrying after sleep of 2000


org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1


At Org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy (hbaserpc.java:424)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 865)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 881)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locaterootregion (HConnectionManager.java:936)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:581)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:557)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregioninmeta (HConnectionManager.java : 631)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:590)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.relocateregion (HConnectionManager.java:563)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.getregionlocation (HConnectionManager.java:407 )


At Org.apache.hadoop.hbase.client.ServerCallable.instantiateServer (servercallable.java:57)


At Org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer (scannercallable.java:62)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.getregionserverwithretries ( hconnectionmanager.java:988)


At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:55)


At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:28)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.listtables (hconnectionmanager.java:433)


At Org.apache.hadoop.hbase.client.HBaseAdmin.listTables (hbaseadmin.java:127)


At Org.apache.hadoop.hbase.generated.master.master_jsp._jspservice (master_jsp.java:125)


At Org.apache.jasper.runtime.HttpJspBase.service (httpjspbase.java:97)


At Javax.servlet.http.HttpServlet.service (httpservlet.java:820)


At Org.mortbay.jetty.servlet.ServletHolder.handle (servletholder.java:502)


At Org.mortbay.jetty.servlet.ServletHandler.handle (servlethandler.java:363)


At Org.mortbay.jetty.security.SecurityHandler.handle (securityhandler.java:216)


At Org.mortbay.jetty.servlet.SessionHandler.handle (sessionhandler.java:181)


At Org.mortbay.jetty.handler.ContextHandler.handle (contexthandler.java:766)


At Org.mortbay.jetty.webapp.WebAppContext.handle (webappcontext.java:417)


At Org.mortbay.jetty.handler.ContextHandlerCollection.handle (contexthandlercollection.java:230)


At Org.mortbay.jetty.handler.HandlerWrapper.handle (handlerwrapper.java:152)


At Org.mortbay.jetty.Server.handle (server.java:324)


At Org.mortbay.jetty.HttpConnection.handleRequest (httpconnection.java:534)


At Org.mortbay.jetty.httpconnection$requesthandler.headercomplete (httpconnection.java:864)


At Org.mortbay.jetty.HttpParser.parseNext (httpparser.java:533)


At Org.mortbay.jetty.HttpParser.parseAvailable (httpparser.java:207)


At Org.mortbay.jetty.HttpConnection.handle (httpconnection.java:403)


At Org.mortbay.io.nio.SelectChannelEndPoint.run (selectchannelendpoint.java:409)


At Org.mortbay.thread.queuedthreadpool$poolthread.run (queuedthreadpool.java:522)


2010-01-13 18:34:18,500 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:18,501 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:18,501 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:20,501 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:20,506 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:20,506 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:20,506 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:22,507 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:22,511 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:22,511 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:24,516 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:24,516 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:24,516 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:26,517 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:26,521 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:26,522 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:26,522 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:28,522 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:28,526 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:28,527 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:28,527 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:locateregioninmeta Attempt 0 of 3 failed; Retrying after sleep of 2000


org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1


At Org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy (hbaserpc.java:424)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 865)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 881)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locaterootregion (HConnectionManager.java:936)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:581)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:557)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregioninmeta (HConnectionManager.java : 631)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.locateregion (HConnectionManager.java:590)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.relocateregion (HConnectionManager.java:563)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.getregionlocation (HConnectionManager.java:407 )


At Org.apache.hadoop.hbase.client.ServerCallable.instantiateServer (servercallable.java:57)


At Org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer (scannercallable.java:62)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.getregionserverwithretries ( hconnectionmanager.java:988)


At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:55)


At Org.apache.hadoop.hbase.client.MetaScanner.metaScan (metascanner.java:28)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.listtables (hconnectionmanager.java:433)


At Org.apache.hadoop.hbase.client.HBaseAdmin.listTables (hbaseadmin.java:127)


At Org.apache.hadoop.hbase.generated.master.master_jsp._jspservice (master_jsp.java:125)


At Org.apache.jasper.runtime.HttpJspBase.service (httpjspbase.java:97)


At Javax.servlet.http.HttpServlet.service (httpservlet.java:820)


At Org.mortbay.jetty.servlet.ServletHolder.handle (servletholder.java:502)


At Org.mortbay.jetty.servlet.ServletHandler.handle (servlethandler.java:363)


At Org.mortbay.jetty.security.SecurityHandler.handle (securityhandler.java:216)


At Org.mortbay.jetty.servlet.SessionHandler.handle (sessionhandler.java:181)


At Org.mortbay.jetty.handler.ContextHandler.handle (contexthandler.java:766)


At Org.mortbay.jetty.webapp.WebAppContext.handle (webappcontext.java:417)


At Org.mortbay.jetty.handler.ContextHandlerCollection.handle (contexthandlercollection.java:230)


At Org.mortbay.jetty.handler.HandlerWrapper.handle (handlerwrapper.java:152)


At Org.mortbay.jetty.Server.handle (server.java:324)


At Org.mortbay.jetty.HttpConnection.handleRequest (httpconnection.java:534)


At Org.mortbay.jetty.httpconnection$requesthandler.headercomplete (httpconnection.java:864)


At Org.mortbay.jetty.HttpParser.parseNext (httpparser.java:533)


At Org.mortbay.jetty.HttpParser.parseAvailable (httpparser.java:207)


At Org.mortbay.jetty.HttpConnection.handle (httpconnection.java:403)


At Org.mortbay.io.nio.SelectChannelEndPoint.run (selectchannelendpoint.java:409)


At Org.mortbay.thread.queuedthreadpool$poolthread.run (queuedthreadpool.java:522)


2010-01-13 18:34:28,547 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:28,547 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:28,547 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:30,547 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:30,552 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:30,552 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:30,552 DEBUG org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:root Region location Changed. Sleeping.


2010-01-13 18:34:31,575 INFO org.apache.hadoop.hbase.master.servermanager:5 region servers, 0 dead, average load 0.16666 666666666666


2010-01-13 18:34:32,552 DEBUG Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers:wake. Retry finding Root region.


2010-01-13 18:34:32,557 DEBUG Org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:Read znode/hbase/ Root-region-server got 10.37.17.248:60020


2010-01-13 18:34:32,558 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:RegionManager.rootScanner scanning meta Region {server:10.37.17.248:60020, regionname:-root-,, 0, Startkey: <>}


2010-01-13 18:34:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner:All 0. META. Region (s) scanned


2010-01-13 18:34:32,741 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:34:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:Scan ROOT Region


org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1


At Org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy (hbaserpc.java:424)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 865)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 881)


At Org.apache.hadoop.hbase.master.BaseScanner.scanRegion (basescanner.java:156)


At Org.apache.hadoop.hbase.master.RootScanner.scanRoot (rootscanner.java:54)


At Org.apache.hadoop.hbase.master.RootScanner.maintenanceScan (rootscanner.java:79)


At Org.apache.hadoop.hbase.master.BaseScanner.chore (basescanner.java:136)


At Org.apache.hadoop.hbase.Chore.run (chore.java:68)


2010-01-13 18:35:31,575 INFO org.apache.hadoop.hbase.master.servermanager:5 region servers, 0 dead, average load 0.16666 666666666666


2010-01-13 18:35:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:RegionManager.rootScanner scanning meta Region {server:10.37.17.248:60020, regionname:-root-,, 0, Startkey: <>}


2010-01-13 18:35:32,741 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:35:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:Scan ROOT Region


org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1


At Org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy (hbaserpc.java:424)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 865)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 881)


At Org.apache.hadoop.hbase.master.BaseScanner.scanRegion (basescanner.java:156)


At Org.apache.hadoop.hbase.master.RootScanner.scanRoot (rootscanner.java:54)


At Org.apache.hadoop.hbase.master.RootScanner.maintenanceScan (rootscanner.java:79)


At Org.apache.hadoop.hbase.master.BaseScanner.chore (basescanner.java:136)


At Org.apache.hadoop.hbase.Chore.run (chore.java:68)


2010-01-13 18:35:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner:All 0. META. Region (s) scanned


2010-01-13 18:36:31,575 INFO org.apache.hadoop.hbase.master.servermanager:5 region servers, 0 dead, average load 0.16666 666666666666


2010-01-13 18:36:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner:RegionManager.rootScanner scanning meta Region {server:10.37.17.248:60020, regionname:-root-,, 0, Startkey: <>}


2010-01-13 18:36:32,741 INFO Org.apache.hadoop.ipc.HbaseRPC:Server at/10.37.17.248:60020 could not is reached after 1 tr IES, giving up.


2010-01-13 18:36:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner:Scan ROOT Region


org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1


At Org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy (hbaserpc.java:424)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 865)


At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 881)


At Org.apache.hadoop.hbase.master.BaseScanner.scanRegion (basescanner.java:156)


At Org.apache.hadoop.hbase.master.RootScanner.scanRoot (rootscanner.java:54)


At Org.apache.hadoop.hbase.master.RootScanner.maintenanceScan (rootscanner.java:79)


At Org.apache.hadoop.hbase.master.BaseScanner.chore (basescanner.java:136)


At Org.apache.hadoop.hbase.Chore.run (chore.java:68)

At this point, hbase can enter the shell, but cannot perform any action. So again when I tried to close hbase, I found that I couldn't stop master, the "." Behind "Stop Master". There are so many and still cannot stop the master node. So we had to force kill master. HBase just hung up ...

Cause analysis of HBase reboot after reboot:
after a variety of troubleshooting, I finally tried to use Netstat-an to view the port occupancy situation when I found:
On the WAMDM1 node, the Regionserver occupies 60020 ports for: 127.0.0.1:60020
On the WAMDM2 node, the Regionserver occupies 60020 ports for: 10.37.17.249:60020
I feel quite strange, then check the/etc/hosts file, and indeed found in WAMDM1 and WAMDM2 under the Hosts file different. The contents of the Hosts file in WAMDM1 are:

The code is as follows	Copy Code
127.0.0.1 WAMDM1 localhost.localdomain localhost 10.37.17.248 WAMDM1.ruc.edu.cn WAMDM1 10.37.17.249 WAMDM2.ruc.edu.cn WAMDM2 10.37.17.250 WAMDM3.ruc.edu.cn WAMDM3 10.37.17.251 WAMDM4.ruc.edu.cn WAMDM4 10.37.17.252 WAMDM5.ruc.edu.cn WAMDM5

Attention to the first line. When we use configuration hadoop/hbase, we often use host names instead of IP, but on WAMDM1 machines, WAMDM1 is mapped to 127.0.0.1, so the communication between master and Regionserver is wrong. That's why we often see the following information in the log as well as in the error message:

The code is as follows

Copy Code

The Server at/10.37.17.248:60020 could is reached after 1 tries and giving up. org.apache.hadoop.hbase.client.RetriesExhaustedException:Failed Setting up proxy to/10.37.17.248:60020 Attempts=1 At Org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy (hbaserpc.java:424) At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 865) At Org.apache.hadoop.hbase.client.hconnectionmanager$tableservers.gethregionconnection (HConnectionManager.java : 881) At Org.apache.hadoop.hbase.master.BaseScanner.scanRegion (basescanner.java:156) At Org.apache.hadoop.hbase.master.RootScanner.scanRoot (rootscanner.java:54) At Org.apache.hadoop.hbase.master.RootScanner.maintenanceScan (rootscanner.java:79) At Org.apache.hadoop.hbase.master.BaseScanner.chore (basescanner.java:136) At Org.apache.hadoop.hbase.Chore.run (chore.java:68)

Solution that cannot be started after hbase reboot:
so I changed the/etc/hosts file for all the machines to the following configuration:

code is as follows

copy code

127.0.0.1 localhost 10.37.17.248 WAMDM1.ruc.edu.cn WAMDM1 10.37.17.249 WAMDM2.ruc.edu.cn WAMDM2 10.37.17.250 WAMDM3.ruc.edu.cn WAMDM3 10.37.17.251 WAMDM4.ruc.edu.cn WAMDM4 10.37.17.252 WAMDM5.ruc.edu.cn WAMDM5 # The Following lines are desirable for IPV6 capable hosts :: 1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

At the same time, to be on the safe side, I also joined the stop-hbase.sh to stop regionservers (although there was no need to stop regionservers on the web, and there was evidence of a bug in the script, but it was not a problem to change the test):

The code is as follows	Copy Code
"$bin"/hbase-daemons.sh--config "${hbase_conf_dir}"--hosts "${hbase_regionservers}" Stop Regionserver

This command needs to be added before stop master. As for the possibility of not joining this stop regionservers command, I will test it further in the future.

Through the above changes, there are two 127.0.0.1 regionserver problems, as well as the hbase restart on the hanging problem is thoroughly resolved!