Highly available management and monitoring of distributed database Hbase (II.)

Figure 5. Biginsights Web Management Interface

Click on the "Cluster Status" page in the biginsights Web admin interface to achieve http://www.aliyun.com/zixun/aggregation/13713.html ">hbase, Zookeeper and other module status monitoring, starting and stopping.

The Cluster Status page lists the running state of services for all components in the Biginsights cluster, respectively. You can see that all services are currently in a normal state of operation. Select "HBase" to view the service information for the HBase cluster, as shown in Figure 8, which lists HBase's brief information, including HBase cluster status (start/stop), HBase Master node and process number, average Load number, HBase cluster-surviving Number of Region servers, number of dead Region servers, etc.

Figure 6. HBase Service Status Monitoring

Figure 7. HBase Master Service Status monitoring and shutdown

Figure 8. HBase Region Servers Service status monitoring and shutdown

In addition, Biginsights provides a unified command-line interface to manage cluster components. Users can easily use the command line to check the health status of the HBase cluster, add and remove HBase nodes, start, stop a service, or query the status of the service.

Check the health status of the HBase component

You can verify the health status of the HBase cluster by using the following command on the Biginsights Management node:

$/bin/sh $BIGINSIGHTS _home/bin/healthcheck.sh HBase

Synchronizing HBase Profiles

As a Cluster Administrator, it is often necessary to modify the configuration parameters to tune the cluster performance of HBase, or to modify the associated configuration, all of which require modification of the HBase configuration file, such as Hbase-site.xml. Regionservers, hbase-env.sh, hadoop-metrics.properties, or log4j.properties,biginsights cluster Management provides a unified profile synchronization function for each component, To ensure consistency in the configuration of the unified component on all nodes. You can synchronize the HBase configuration file with the following command:

1. You can update hbase configuration files in the $BIGINSIGHTS _home/hdm/components/hbase/conf/directory of the Biginsights management node

2. On the Biginsights Management node, synchronize all node HBase configuration files by running the following command:

$/bin/sh $BIGINSIGHTS _home/bin/syncconfig.sh

HBase additions and deletions hbase nodes

The Biginsights platform supports the ability to delete HBase nodes.

1. Add nodes

You can add one or more nodes to the HBase cluster by running the following command on the Biginsights Management node:

$/bin/sh $BIGINSIGHTS _home/bin/addnode.sh hbase node1-type=master

$/bin/sh $BIGINSIGHTS _home/bin/addnode.sh hbase node1-type=regionserver

Where node1 can be an IP address or a hostname.

2. Delete node

You can remove one or more nodes from the HBase cluster by running the following command on the Biginsights Management node, and deleting the HBase node only deletes the HBase service and does not delete any data.

$/bin/sh $BIGINSIGHTS _home/bin/removenode.sh HBase

Where node1 can be an IP address or a hostname.

Kai-Stop and view HBase services

You can start the HBase Cluster service by running the following command on the Biginsights Management node:

$/bin/sh $BIGINSIGHTS _home/bin/start.sh HBase

You can stop the HBase Cluster service by running the following command on the Biginsights Management node:

$/bin/sh $BIGINSIGHTS _home/bin/stop.sh HBase

View HBase Services:

You can view the HBase Cluster service status by running the following command on the Biginsights Management node:

$/bin/sh $BIGINSIGHTS _home/bin/status.sh HBase

HBase Master Multi-node high availability support

We can start multiple master nodes in the HBase cluster, where the first startup node serves as active (active) master, while the others are backup master nodes; in fact, HBase through zookeeper To save the active master node information, once the active master node loses its connection due to a hardware failure or power outage, backup master receives the information through zookeeper and elects to generate a new active master service. The next step is to describe how to start multiple Master through biginsights and monitor the status.

1. Add/start/monitor HBase Master via command line mode

A) login to biginsights management node to view current HBase cluster status

[biadmin@bdvm070 ~]$ status.sh HBase

[INFO] Progress-status HBase

[INFO] @bdvm070. Svl.ibm.com-hbase-master (Active) started, PID 18157

[INFO] @bdvm071. Svl.ibm.com-hbase-regionserver started, PID 12072

[INFO] Deployer-hbase Service started

[INFO] progress-100%

[INFO] Deploymanager-status; Succeeded RS: [HBase]; Consumes:9699ms

B add bdvm071 as alternate Master node

[biadmin@bdvm070 ~]$ addnode.sh hbase bdvm071.svl.ibm.com-type=master

[INFO] Deploymanager-check SSH setup on [bdvm071.svl.ibm.com]

[INFO] @bdvm071. Svl.ibm.com-ok, password-less SSH super-delegates setup.

...

[INFO] Deploymanager-add hbase nodes; Succeeded RS: [HBase]; Consumes:0ms

C) Start the alternate Master node

[biadmin@bdvm070 ~]$ start.sh HBase

[INFO] Progress-start HBase

[INFO] Deployer-make throaty Zookeeper Service is started unreported start HBase service

[INFO] @bdvm070. Svl.ibm.com-hbase-master (Active) already running, PID 18157

[INFO] @bdvm071. Svl.ibm.com-hbase-master (Backup) started

[INFO] @bdvm071. Svl.ibm.com-hbase-regionserver already running, PID 12072

[INFO] Deployer-hbase Service started

[INFO] progress-100%

[INFO] Deploymanager-start; Succeeded RS: [HBase]; Consumes:14110ms

D) Re-view the HBase cluster status

[biadmin@bdvm070 ~]$ status.sh HBase

[INFO] Progress-status HBase

[INFO] @bdvm070. Svl.ibm.com-hbase-master (Active) started, PID 18157

[INFO] @bdvm071. Svl.ibm.com-hbase-master (Backup) started

[INFO] @bdvm071. Svl.ibm.com-hbase-regionserver started, PID 12072

[INFO] Deployer-hbase Service started

[INFO] progress-100%

[INFO] Deploymanager-status; Succeeded RS: [HBase]; Consumes:14382ms

E Stop Current active Master node and view HBase cluster status

[biadmin@bdvm070 ~]$ $HBASE _home/bin/hbase-daemon.sh Stop Master

Stopping master.

[biadmin@bdvm070 ~]$ status.sh HBase

[INFO] Progress-status HBase

[INFO] @bdvm070. Svl.ibm.com-hbase-master stopped

[INFO] @bdvm071. Svl.ibm.com-hbase-master (Active) started, PID 19840

[INFO] @bdvm071. Svl.ibm.com-hbase-regionserver started, PID 12072

[INFO] Deployer-hbase Service started

[INFO] progress-100%

[INFO] Deploymanager-status; Succeeded RS: [HBase]; Consumes:52069ms

As we can see from the above, we can add backup master at any time in the Biginsights cluster and perform master's shutdown/monitoring through the command line. It is also important to note that Backup Master detects active master downtime and switches to active master for a certain amount of time, so avoid actively stopping the active master process during the actual cluster usage. The following sequence of operations in the Biginsights Web Management page monitoring.

2. Biginsights WEB Admin page monitor HBase cluster status

A) add standby master before HBase master run state

Figure 9. Add Standby Master Pre HBase Master run status

B add standby Master and start HBase Master run state

Figure 10. HBase Master run status after adding standby master

C Stop current active master HBase Master run state

Figure 11. HBase Master Run status after stopping active master

D HBase Master Running after restarting the stopped master node (since active master is now available, the restarted master node will be the standby master)

Figure 12. HBase Master runs after restarting the stopped master node

You can also easily start/stop the designated Master node through the Biginsights Web Administration page, and please refer to the relevant chapters for more details.

Reverse proxy HBase UI

Reverse Proxy is the reverse proxy tool provided by biginsights, which enables reverse proxy connection support for the service components of the UI interface in Biginsights. (Note: This feature is available only in the Biginsights Enterprise version.) ）

Figure 13. Reverse Proxy Admin HBase page

For example, after opening the Reverse Proxy Administration page, using the URL and Alias mapping features provided by Reverse Proxy, you can easily redirect the service UI through intuitive links.

Through the Reverse Proxy page, click "Hbase-master" to open the HBase Master Admin page directly.

Figure 14. Open HBase Master Administration page

Through Reverse Proxy page, click "Hbase-regionserver" can directly open hbase regionserver admin page.