Build a Scalable Distributed Transaction Queue Based on HBase

Build a Scalable Distributed Transaction Queue Based on HBase

A Real-time stream processing framework usually requires two infrastructures: processor and queue. The processor reads the event from the queue and executes the user's processing code. To continue processing the result, the processor also writes the event to another queue. Queues are provided and managed by the framework. The queue acts as a buffer between processors to transmit data and events, so that the processor can operate and expand independently. For example, a web Service may access a log processing application as follows:

 

 

The main difference between frameworks is the queue semantics. The differences are as follows:

• Scheduling assurance mechanism: at least once, at most once, only once.

• Disaster Tolerance Mechanism: failure is transparent to users and automatic recovery.

• Availability: data can be saved and restarted when an error occurs.

• Scalability: the limitation of product/user increase.

• Performance: throughput and latency of queue operations.

We want to provide a real-time stream processing framework with dynamic scalability, strong consistency, and one-time transaction mechanism on the open-source Cask Data Application Platform (CDAP). Under the protection of this powerful mechanism, developers can operate data in any form without worrying about inconsistency and potential rework and failure. It helps developers establish their big data applications without the background of distributed systems. In addition, if necessary, you can disable this powerful protection mechanism in exchange for high performance. It is always easier to use than other methods.

We want to provide a real-time stream processing framework with dynamic scalability, strong consistency, and one-time transaction mechanism on the open-source Cask Data Application Platform (CDAP). Under the protection of this powerful mechanism, developers can operate data in any form without worrying about inconsistency and potential rework and failure. It helps developers establish their big data applications without the background of distributed systems. In addition, if necessary, you can disable this powerful protection mechanism in exchange for high performance. It is always easier to use than other methods.

Scalable queue

There are two basic operations for a queue: Join and exit. The producer writes the message to the queue header (into the queue), and the consumer reads the data (out of the queue) from the end of the queue ). If, as a whole, you can add more members to the queue quickly enough to add more consumers, we say this queue is scalable.

Ideally, expansion is linear, which means that two times of producers/consumers will generate two queues/queues, and the growth will only be limited by the size of the cluster. To support linear expansion of producers, the queue requires a storage system and linear expansion of the number of current writers. To cope with the linear expansion of consumers, a queue can be partitioned. For example, a consumer can only process a piece of data in the queue.

Another aspect of queue scaling is that it should be horizontally scalable. This means that the maximum queue performance can be improved by adding cluster nodes. This is very important. It can ensure that the queue is not limited by the current cluster size and can be expanded based on data growth.

Partition HBase queue

We chose Apache HBase as the storage layer of the queue. It designs and optimizes row data with strong consistency and scalability. It provides excellent concurrent Write Performance and sequential scanning to support partition consumers. We use HBase Coprocessors for efficient scanning filtering and queue cleaning. To use one-time semantics on the queue, we use Tephra's to provide Transmission Support for HBase.

Producers and consumers have operational independence. Each producer queues through Hbase puts batch processing, and the consumer queues through Hbase Scans. There is no correlation between the number of producers and consumers. They can be separated.

This queue has the concept of a consumer group. A consumer group is a set of consumers divided by the same keywords. In this way, each event published to the queue will be consumed by the consumers in the consumer group. With consumer groups, you can use different keywords to divide the same queue, or expand through the operational features of data. According to the preceding access log analysis example, the producer and consumer groups may look like this:

 

 

For Log Parser, two producers are running, and they write data to the queue concurrently. On the consumer side, there are two consumer groups. The Unique User Counter group has two consumers who use UserID as the key word for grouping (Queue. The Page View Counter group has three consumers, which use PageID as the key word for partitioning (Queue.

Queue row value format

When an event is published by a producer and one or more consumers receive a message, we write the event to one or more rows in the HBase table, this record is designed to apply to each consumer group. The effective load of the event and metadata are stored in an independent column. The row value is in the following format:

 

The two interesting parts are that the row value is the partition ID and the whole ID. The partition ID is provided to the consumer by specifying the row value prefix. The consumer is only allowed to read data, and uses the prefix to scan when leaving the queue. The partition ID consists of a consumer group ID and a consumer ID. The producer calculates the shard ID of each consumer group and writes the row into the queue.

The Entry ID in the row keyword contains the transaction information. It consists of the producer transaction write pointer triggered by Tephra and the one-way growth counter. This counter is generated by the local producer. At the same time, the counter must make the row keyword unique for the event, because the producer can add multiple events to the queue in the same transaction.

When the queue is output, the counter uses the transaction write pointer to determine whether the queue entry has been committed and whether it can be consumed. The combination of transaction write pointers and counters makes the row keywords always unique. This allows the producer to perform independent operations without writing conflicts.

To generate a Partition ID (Partition ID), the producer needs to know the size and Partition Keywords of each consumer group. When the application starts and the group size changes, the consumer group information is recorded.

Change producers and consumers

Adding or reducing producers is direct because each producer operates independently. Adding or decreasing producer processes can meet this requirement. However, when the size of a consumer group needs to be changed, coordination is required to correctly update the information of the consumer group. You can use the following chart to summarize the required steps:

 

Since the pause and restore operations are coordinated by Apache ZooKeeper and executed in parallel, they are two very fast operations. For example, the Web access log analysis application we mentioned earlier may change the consumer group information as follows:

Based on the design of this queue, the performance of the inbound and outbound queues is comparable to that of the individual batch HBase Puts and HBase Scans, which also brings about the communication overhead with the Tephra server. This overhead can be greatly reduced by batch processing multiple events in the same service.

Finally, to avoid "hotspotting", we split the HBase table in advance based on the cluster size. At the same time, we used salting on the row key) to better allocate writes. Otherwise, the row keywords will be continuous as the business processing write pointer is added monotonically.

 

Performance Value

 

We have tested the performance on a small 10-node HBase cluster and the results are impressive. Using a 1 K byte load and taking 500 events as a batch size, we have produced and consumed 100 K events/s of throughput, three producers and 10 Consumers are running. We also observe that throughput increases linearly when we increase consumers and consumers: for example, when we double the number of producers and consumers, throughput increases to 200 K events/s.

 

With the help of HBase, combined with best practices, we have successfully created a linearly scalable, distributed transaction queue system. At the same time, this system is used in CDAP to provide a real-time stream processing framework: Dynamic scalability, strong consistency, and one-time delivery transmission guarantee.

Hadoop + HBase cloud storage creation summary PDF

Regionserver startup failed due to inconsistent time between HBase nodes

Hadoop + ZooKeeper + HBase cluster configuration

Hadoop cluster Installation & HBase lab environment setup

HBase cluster configuration based on Hadoop cluster'

Hadoop installation and deployment notes-HBase full distribution mode installation

Detailed tutorial on creating HBase environment for standalone Edition

HBase details: click here
HBase: click here

Scalable Distributed Transactional Queues on HBase

This article permanently updates the link address: