NetEase Video Cloud technology sharing: hbase High availability principle and practice

NetEase Video Cloud is a cloud-based distributed multimedia processing cluster and professional audio and video technology designed by NetEase to provide stable, smooth, low-latency, high-concurrency video streaming, recording, storage, transcoding and VOD, such as the PAAs service, online education, telemedicine, entertainment show, Online finance and other industries and enterprise users only through simple development can create online audio and video platform. Now, NetEase video cloud Technical experts to share a technical article:HBase High Availability principle and practice.

Objective

There was a little problem with a set of online hbase, which caused the HBase Cluster service to stop for 2 hours, resulting in a service exception for applications that use the set of HBase as a data store. In the troubleshooting of problems, we can not help but also thinking, the future of similar problems to do? How to avoid this problem? Used to MySQL, and then thought that HBase is the same as MySQL, there is a high-availability program?

The answer is of course yes, almost all databases (whether relational or distributed), using the WAL approach to ensure data recovery when the service is abnormal, hbase is also through the Wal to ensure that data is not lost. HBase writes the data before it writes the Hlog,hlog, and the high availability of hbase is achieved by Hlog.

Advanced

HBase is a distributed system with no single point of failure, and the upper layer (hbase layer) and the bottom (HDFs layer) Ensure the availability of the service through certain technical means. Upper Hmaster are generally high-availability deployments, and regionserver if there is a outage, the cost of region migration is not small, generally at the millisecond level, so the impact on the application is also limited; the underlying storage relies on HDFS, and the data itself has 3 copies by default. Multiple copy redundancy is achieved on the datastore, and the single point of failure of Namenode after Hadoop 2.0 is eliminated. So, is it necessary to have a highly available configuration for a system that does not have a single point of failure and has multiple copies of the data? Will it cause a great waste of resources?

Whether a highly available deployment is necessary depends on the importance of the service, and here's a quick introduction to what's going to happen without a highly available HBase service:

Database Manager error, irreversible DDL operation

Regardless of the database, DDL operations need to be cautious when executing, possibly a simple drop operation that will cause all data to be lost and unrecoverable, as is the case with HBase, and if the administrator accidentally dropped a table, the data for that table would be lost.

Offline Mr Consumes too much resources, causing online service to be impacted

HBase after so many years of development, is no longer only suitable for offline business data storage analysis platform, many of the company's online business has been migrated to HBase. But inevitably in these data to do some statistical analysis of the operation, large Mr Run, there will be a lot of resource consumption, may affect the online business.

Other unforeseen circumstances

such as core switch failure, room blackout, etc. will cause hbase service interruption

For the above problems, you can configure the high of HBase to address:

Non-reversible DDL problem

The high availability of hbase does not support DDL operations, in other words, DDL operations on Master do not affect data on slave, so even with DDL operations on master, the data on slave remains unchanged. This is very different from MySQL, MySQL DDL can be copied through the statement format of Binlog.

Offline Mr Affects online business issues

The biggest benefit of high availability is the ability to read and write separation, offline Mr can run directly on the slave, Master continues to provide write service, so it will not affect the online business, of course, hbase high-availability replication is asynchronous, slave on the Mr Analysis, The data may be slightly delayed.

Unexpected situation

For unexpected situations such as core switch failures, power outages, slave or cross-machine room deployments can solve this situation.

Based on the above reasons, if it is the core service, for the availability requirements are very high, can build hbase high availability to ensure higher availability of the service, in HBase Master exception, simply transfer traffic to slave, you can complete the failover, to ensure that the service runs properly.

Principle

HBase High Availability guarantees rapid failover in the event of an exception. Let's take a look at the implementation of the HBase high availability first, and look at an official picture:

HBase Replication

You need to declare that the replication of HBase is in column family, and each column family can be set for replication.

, a master corresponds to 3 slave,master on each regionserver has a copy of the Hlog, in the case of opening replication, Each regionserver opens a thread that reads the Hlog on the Regionserver and sends it to each slave,zookeeper to save the location of the Hlog that is currently sent. The use of asynchronous communication between master and slave ensures that the performance on Master is not affected by slave. Use zookeeper to save the location where the Hlog has been sent, mainly considering the location of the last replication if you re-establish the replication during the slave replication process if a problem occurs.

HBase Replication Steps

HBase client writes data to master

Corresponding Regionserver after writing Hlog return client request

At the same time replication thread poll Hlog found new data, sent to slave

Slave return to master after processing data

Master receives the return information of slave and marks the Hlog location that has been sent to slave in zookeeper

Note: in the replication, master and slave configuration is not necessarily the same, such as Master can have 3 units on the Regionserver,slave is not necessarily 3 units, The number of regionserver on the slave can be different, and how the data is distributed within hbase will be handled.

Kinds

HBase uses Hlog for data replication, so what are the different kinds of replication relationships that HBase supports?

In terms of replication mode, HBase supports both master-slave and Main-master replication modes, which are often said to be master-slave, master-master replication.

Master-slave

Master-slave replication is simple, and all data written on the master cluster will be synchronized to the slave.

Master-master

Master-master replication is similar to Master-slave, the main difference being that in Master-master replication, two master status is the same and can be read and written.

Since master-master two master can write, what happens if a situation arises: two Master writes to the same rowkey of the same column family on the same table?

Create ' t ', {name=> ' CF ', replication_scope=> ' 1 '}

Master1 Master2

Put ' t ', ' R1 ', ' cf ', ' aaaaaaaaaaaaaaa ' put ' t ', ' R1 ', ' cf ', ' bbbbbbbbbbbbbbb '

As above operation, the CF column of T on Master1 writes Rowkey to R1,value as AAAAAAAAAAAAAAA data, and the CF column of T on Master2 writes Rowkey to R1, and value is BBBBBBBBBBBBBBB data , because it is master-master replication, Master1 and Master2 write the data at the same time as the update is sent to the other side, so the final data becomes:

Master1 Master2

Rowkey Value Rowkey Value

R1 BBBBBBBBBBBBBBB R1 AAAAAAAAAAAAAAA

As you can see from the table above, the CF-Clustered rowkey on the Master1 and Master2 are inconsistent on both sides of the R1 data.

so, in doing Master-master when highly available, ensure that the tables written on both sides are different, which prevents the data inconsistency problem.

Abnormal

When HBase is copied, it is sent by Regionserver to open the replication thread for Hlog, so how does hbase handle when one of the Regionserver has an exception? There are two different situations that need to be distinguished, that is, the regionserver exception on master and the regionserver exception on slave.

Slave on Regionserver Exception

For this kind of exception hbase processing is relatively simple, there is a regionserver exception on the slave, the regionserver is directly marked as an abnormal state, all subsequent updates will not be sent to the Regionserver, Slave will re-select a regionserver to receive this part of the data.

Master on Regionserver Exception

Master on Regionserver exception, because hlog all through regionserver to open the replication thread to send, if the Regionserver an exception, this time, The hlog that belong to the regionserver have no related processing thread, so what do you do with this part of the data?

There is an Regionserver exception on master, and other regionserver will attempt to lock the information of the regionserver in zookeeper, of course, this operation is mutually exclusive, At the same time only one regionserver can acquire the lock, then, the Hlog information will be copied to their own directory, so that the abnormal Regionserver hlog information transfer, through the new regionserver to send hlog information to slave.

Master Regionserver crash

Operation

The above is the hbase high availability of the theoretical implementation and exception handling problems, the following hands-on, how to configure a hbase replication (assuming that two sets of hbase system has been deployed, and in the configuration file has been opened replication configuration), First try to configure the high availability under Master-slave mode:

Select a system as master and another set as slave

On master, add the replication relationship via the Add_peer command, as follows

Add_peer ' 1 ', "db-xxx.photo.163.org:2181:/hbase"

To create a new table T on master, the table has a column cluster named CF, and the column cluster opens replication, as follows:

Create ' t ', {name=> ' CF ', replication_scope=> ' 1 '}

The above Replication_scope value needs to correspond with the corresponding in step 2

In slave to establish the same table (HBase does not support DDL replication), in Master-slave mode, slave does not need to turn on replication, as follows:

Create ' t ', {name=> ' CF '}

In this way, we have completed a high-availability build of the entire Master-slave mode, and later can insert a record on master with a put operation to see if the record will be copied on the slave and the final result is as follows:

Operations on Master

Results on slave

The above results show that after the replication relationship is added, Master inserts ROWKEY=R1, value= ' aaaaaaaaa ' records, slave can get the record, Master-slave mode data replication succeeds.

Next we look at the replication of the Master-master mode, when configured with the Master-slave mode, when you add a copy relationship on Master, you need to add a copy relationship to the other master, and the cluster_id must be the same on both sides. And when you build a table on another master, you need to add the replication_scope=> ' 1 ' Configuration of the cluster, and the result is as follows:

Operation on Master1

Operation on Master2

The above results show that the Master-master replication relationship has been added, a record is inserted on the Master1 rowkey=r1, value= "Aaaaaaaaaa", Master2 on a scan operation found that the record has been copied to Master2, Then we add a record on the Master2 rowkey=r2, value= ' BBBBBBBBBBBB ', view the data on the Master1, the record has also been copied to Master2, master-master the replication validation of the mode is successful.

NetEase Video Cloud technology sharing: hbase High availability principle and practice