Netease video Cloud technology sharing: Principles and Practices of high availability of HBase

Netease video cloud is a cloud computing-based Distributed Multimedia Processing cluster and professional audio and video technologies built by Netease, provides stable, smooth, low-latency, and high-concurrency PAAS services for live video, recording, storage, transcoding, on-demand audio and video, online education, telemedicine, and entertainment.

Netease video cloud is a cloud computing-based Distributed Multimedia Processing cluster and professional audio and video technologies built by Netease, provides stable, smooth, low-latency, and high-concurrency PAAS services for live video, recording, storage, transcoding, on-demand audio and video, online education, telemedicine, and entertainment.

Netease video cloud is a cloud computing-based Distributed Multimedia Processing cluster and professional audio and video technologies built by Netease, it provides stable, smooth, low-latency, and high-concurrency PAAS services for live video, recording, storage, transcoding, on-demand audio and video, online education, telemedicine, entertainment shows, online finance, and other industries and enterprise users can create an online audio and video platform through simple development. Now, Netease video cloud technical experts share a technical article on the principle and practice of high availability of HBase.

Preface

Some time ago there was a small problem with the online HBase, which caused the HBase cluster service to be stopped for two hours, as a result, service exceptions also occur for applications that use this HBase as data storage. In addition to troubleshooting, we can't help but think about it. What should we do if we encounter similar problems in the future? How can we avoid this problem? When I used MySQL, I thought about whether HBase is the same as MySQL and has its high-availability solution?

The answer is certainly yes. Almost all databases (relational or distributed) Adopt the WAL method to ensure data recovery in case of service exceptions, HBase also uses WAL to prevent data loss. HBase writes HLog before writing data. HLog records all data changes. HBase's high availability is also achieved through HLog.

Advanced

HBase is a distributed system without spof. Both the upper layer (HBase layer) and the bottom layer (HDFS layer) use certain technical means to ensure service availability. The upper-layer HMaster is generally deployed in high availability mode. If the RegionServer goes down, the region migration cost is not large and generally completed in milliseconds. Therefore, the impact on applications is limited; the underlying storage depends on HDFS, and the data itself also has three copies by default. The data storage achieves multi-copy redundancy, and the single point of failure of NameNode after Hadoop 2.0 is also eliminated. Therefore, is it necessary for such a system with no single point of failure and redundant data copies to implement high-availability configuration? Will it cause a great waste of resources?

Whether high-availability deployment is necessary depends on the importance of the service. Here we will briefly introduce the problems that will occur when there is no high-availability HBase service:

Database administrators make mistakes and perform irreversible DDL operations

No matter what database, DDL operations must be cautious when executed. A simple drop operation may lead to loss of all data and cannot be recovered. This is also true for HBase, if the Administrator accidentally drops a table, the data in the table will be lost.

Offline MR consumes too much resources and affects online services.

After so many years of development, HBase is no longer a data storage and analysis platform suitable for offline businesses. Many companies' online businesses have also been migrated to HBase. Typical examples include: facebook's Messages system, 360 of search services, and history data of Xiaomi chat. However, it is inevitable to perform statistical analysis operations on the Data. When a large MR runs, it will consume a lot of resources and may affect online services.

Unpredictable situations

For example, HBase service interruption may occur when the core switch fails or the data center loses power.

You can configure HBase to solve the above problems:

Irreversible DDL Problems

High Availability of HBase does not support DDL operations. In other words, DDL operations on the master node do not affect the data on the slave. Therefore, even if DDL operations are performed on the master node, the data on slave remains unchanged. This is very different from MySQL. MySQL DDL can be copied using the statement format Binlog.

Offline MR affects online business issues

The biggest advantage of high availability is that read/write splitting can be performed. Offline MR can be directly run on the slave, and the master continues to provide write services to external users, which will not affect online services, of course, HBase's high-availability replication is asynchronous. When performing MR Analysis on slave, data may be slightly delayed.

Unexpected situation

Server Load balancer can be deployed across racks or data centers in case of core switch faults or power outages.

For the above reasons, if it is a core service with high availability requirements, you can set up a high level of HBase to ensure high availability of the service. In the case of an exception in the HBase Master, you only need to simply switch the traffic to the Slave to complete the Failover and ensure the normal operation of the service.

Principle

HBase High Availability ensures quick failover when exceptions occur. Next, let's take a look at the high availability Implementation of HBase. First, let's take a look at the official figure:

HBase Replication

It should be declared that the replication of HBase is in the Column Family unit, and each Column Family can be set to whether to perform replication.

A Master corresponds to three Slave instances, and each RegionServer on the Master has an HLog. When Replication is enabled, each RegionServer opens a thread to read the HLog on the RegionServer, and sent to each Slave. Zookeeper is used to save the location of the HLog that has been sent. Asynchronous communication is used between the Master and Slave to ensure that the performance of the Master is not affected by the Slave. Use Zookeeper to save the location where HLog has been sent. If a problem occurs during the Slave replication process, re-establish the replication, you can find the last replication location.

HBase Replication procedure

HBase Client writes data to the Master

The corresponding RegionServer returns the Client request after writing the HLog

At the same time, the replication thread polls HLog and finds new data, which is sent to Slave.

After Slave finishes processing the data, it returns it to the Master

The Master receives the Slave return information and marks the HLog location that has been sent to the Slave in Zookeeper.

Note: during replication, the configurations of the Master and Slave are not necessarily the same. For example, three regionservers can be deployed on the Master and not necessarily three on the Slave, the number of regionservers on the Slave can be different. How to distribute the data will be processed internally in HBase.

Type

If HBase replicates data through HLog, what types of replication relationships does HBase support?

In terms of the replication mode, HBase supports Master-Slave and Master-Master replication modes, that is, Master-Slave and Master-Master replication.

Master-Slave

Master-Slave replication is relatively simple. All data written to the Master cluster will be synchronized to the Slave.

Master-Master

Master-Master replication is similar to Master-Slave. The main difference is that in Master-Master replication, both Master nodes have the same status and can be read and written.

Since both Master and Master can write data, in the event of one situation, both Master and slave databases write data to the same rowkey of the same Column Family in the same table, what will happen?

Create 'T', {NAME => 'cf ', REPLICATION_SCOPE => '1 '}

Master1 Master2

Put 'T', 'r1', 'cf ', 'aaaaaaaaaaaaaaa' put 'T', 'r1', 'cf', 'bbbbbbbbbbbbbbbb'

In the preceding operation, the rowkey of the cf column cluster of t on Master1 is r1, the value is aaaaaaaaaaaaaaa, And the rowkey of the cf column cluster of t on Master2 is r1 at the same time, the value is bbbbbbbbbbbbbbb data. Because it is a Master-Master replication, the update will be sent to the other party while writing data on Master1 and Master2, so that the final data will become:

Master1 Master2

Rowkey value

R1 bbbbbbbbbbbbbbbbb r1 aaaaaaaaaaaaaaa

From the above table, we can see that the data in the cf column cluster rowkey of Master1 and Master2 is inconsistent on both sides.

Therefore, when performing Master-Master high availability, make sure that the tables written to both sides are different, which can prevent the above data inconsistency problem.

Exception

During HBase replication, the replication thread is enabled through RegionServer for HLog sending. So how does HBase handle an exception in a RegionServer? There are two differences: RegionServer exception on Master and RegionServer exception on Slave.

RegionServer exception on Slave

HBase is easy to handle this exception. If a RegionServer exception occurs on the Slave, The RegionServer is directly marked as abnormal, and all subsequent updates will not be sent to the RegionServer, slave reselect A RegionServer to receive the data.

RegionServer exception on Master

The RegionServer on the Master node has an exception because HLog is sent by enabling the replication thread on the RegionServer. If the RegionServer encounters an exception, the HLog belonging to the RegionServer does not have a related processing thread. At this time, what should I do with this part of data?

An exception occurs on a RegionServer on the Master node. Other regionservers attempt to lock the information of the RegionServer in zookeeper. Of course, this operation is mutually exclusive. Only one RegionServer can obtain the lock at a time, then, the HLog information will be copied to your own directory, so that the abnormal RegionServer's HLog information is transferred, and the HLog information will be sent to Slave through the new RegionServer.

Master regionserver crash

Operation

The above section describes the high-availability theory implementation and Exception Handling of HBase. Next we will introduce how to configure an HBase Replication (assuming that two HBase systems have been deployed, and the replication configuration has been enabled in the configuration file), first try to configure the Master-Slave mode high availability:

Select one system as the Master and the other as the Slave

Run the add_peer command on the Master node to add a replication relationship.

Add_peer '1', "db-xxx.photo.163.org: 2181:/hbase"

Create a new table t on the Master. The table has a column cluster named cf, and the column cluster enables replication, as shown below:

Create 'T', {NAME => 'cf ', REPLICATION_SCOPE => '1 '}

The value of REPLICATION_SCOPE must correspond to that in step 2.

Create the same table on slave (HBase does not support DDL replication). In master-slave mode, slave does not need to enable replication, as shown below:

Create 'T', {NAME => 'cf '}

In this way, the master-slave mode is highly available. you can insert a record through the put Operation on the master to check whether the record will be copied on the slave, the final result is as follows:

Operations on the Master

Slave result

The above results show that after the replication relationship is added, a record with rowkey = r1 and value = 'aaaaaaaaaaa' is inserted on the Master. This record can be obtained on the slave, data in Master-Slave mode is successfully copied.

Next, let's look at the replication in Master-Master mode. What is different from the Master-Slave mode in configuration is that after the replication relationship is added to the Master, you need to add a replication relationship to another Master, and the cluster_id on both sides must be the same. When creating a table on another Master, you need to add the REPLICATION_SCOPE => '1' configuration of the column cluster, the final result is as follows:

Operations on Master1

Operations on Master2

The above results show that after the Master-Master replication relationship is added, a record rowkey = r1, value = "aaaaaaaaaa" is inserted on Master1 ", the scan operation on Master2 found that the record has been copied to Master2. Then we add a record rowkey = r2, value = 'bbbbbbbbbbbbbbbb' on Master2 to view data on master1, this record has also been copied to Master2, and replication in Master-Master mode is successfully verified.