Three ways of Hbase split and the split process

Split is an important feature in HBase, and hbase is load balanced by assigning data to a certain number of region. A table is assigned to one or more region, which is assigned to one or more regionserver. In the auto split strategy, when a region reaches a certain size, it automatically split into two region. Table is sorted by row key in region, and the row for a row key is stored in only one region, which guarantees the strong consistency of hbase.

There is one or more stroe in a region, and each Stroe corresponds to one column families (column family). A store contains a memstore and 0 or more store files. Each column family is stored separately and accessed separately.

Pre-splitting

When a table has just been created, HBase assigns a region to the table by default. This means that at this time, all the read and write requests will be accessed to the same region of the same regionserver, this time can not achieve the effect of load balancing, the other regionserver in the cluster may be in a relatively idle state. To solve this problem, you can use pre-splitting to create a table and configure it to generate multiple region.

When table initialization is not configured, HBase does not know how to split region, because HBase does not know that the row key should be the starting point for split. If we could probably predict the distribution of the row key, we could use pre-spliting to help us split the region ahead of time. But if we don't predict correctly, it could cause a region to overheat and be centrally accessed, but fortunately we still have auto-split. The best way to do this is to first predict the split point, do pre-splitting, and then let Auto-split handle the load balancer behind it.

HBase comes with two pre-split algorithms, Hexstringsplit and Uniformsplit, respectively. If our row key is prefixed with a hexadecimal string, it is more appropriate to use Hexstringsplit as the pre-split algorithm. For example, we use Hexhash (prefix) as the prefix for row key, where Hexhash is the hash algorithm that finally gets the hexadecimal string. We can also use our own split algorithm.

Under the HBase Shell:

HBase org.apache.hadoop.hbase.util.RegionSplitter pre_split_table hexstringsplit-c 10-f F1

The meaning of-c 10 is that the final region number is 10, and-F F1 to create a column family that is F1.

Execute scan ' Hbase:meta ' to see the meta table,

Only the records of the 2 region of the meta-table (10 areas in total) were intercepted, respectively, and the Rowkey range was the area of ' ~19999999 and 19999999~33333332.

We can also customize the segmentation points, such as using the following command under the HBase Shell:

Create ' t1 ', ' F1 ', {splits = [' 10 ', ' 20 ', ' 30 ', ' 40 ']}

Automatic splitting

When a reion reaches a certain size, he will automatically split to say two region. If our hbase version is 0.94, then there are three automatic split strategies by default, and Constantsizeregionsplitpolicy,increasingtoupperboundregionsplitpolicy Keyprefixregionsplitpolicy.

Prior to version 0.94, Constantsizeregionsplitpolicy was the default and unique split strategy. When the size of a store (corresponding to a column family) is greater than the configuration value ' Hbase.hregion.max.filesize ' (default 10G), region will split automatically.

In version 0.94, Increasingtoupperboundregionsplitpolicy is the default split policy.

In this strategy, the smallest split size is related to the region number of a region server in table, and when the size of the store file is larger than the value of the formula below, it will split, the formula is as follows

Min (r^2 * "Hbase.hregion.memstore.flush.size", "Hbase.hregion.max.filesize")  R for the same table in the same region The number of region in the server.

For example:

Hbase.hregion.memstore.flush.size default value of 128MB.

The default value for Hbase.hregion.max.filesize is 10GB.

• If the initial r=1, then min (128MB,10GB) =128MB, that is, the first flush will trigger a split operation.

• When r=2, Min (2*2*128MB,10GB) =512MB, when a store file size reaches 512MB, it will trigger split.

• So and so on, when the r=9, when the store file reaches 10GB, it will split, that is, when the r>=9, the store file reaches 10GB will split.

Split points are located in the middle of row key in region.

Keyprefixregionsplitpolicy can guarantee that the same prefix row is saved in the same region.

Specifies the number of Rowkey prefix bits divided by the region, by reading the Keyprefixregionsplitpolicy.prefix_length property, which is a numeric type that represents the prefix length, when split is performed, The splitpoint is intercepted at this length. This strategy is more suitable for fixed prefix rowkey. When this property is not set in table, specifies that this policy effect is equivalent to using Increasingtoupperboundregionsplitpolicy.

We can specify the split strategy by configuring Hbase.regionserver.region.split.policy, and we can also write our own split strategy.

Force split

HBase allows the client to force split to execute the following command in the HBase Shell:

Split ' forced_table ', ' B '//Where forced_table is the table to split, ' B ' is the split point

Region Splits Execution Process:

When the region server processes write requests, it writes Memstore and writes to the disk as a store file when the memstore reaches a certain size. This process is called Memstore flush. When the store files accumulate to a certain size, region server performs the ' compact ' operation and synthesizes them into a large file. Each time you perform a flush or compact operation, you will be judged if you need split. When split occurs, two region A and region B are generated, but the parent region data file does not replicate such operations, but region A and region B will have these file references. These reference files are cleaned up the next time the compact operation occurs, and the split operation is not performed when there is a reference file in the region. This place needs to be noted that if there is a reference file in the region, and the write operation is very frequent and concentrated, it may become large, but not split. Because the write operation is more frequent and centralized, but not evenly to each reference file up, so the region has been a reference file, can not be split, this article describes the situation, summed up very well. http://koven2049.iteye.com/blog/1199519

Although the split region operation is determined separately by region server, the split process must work with many other components. Region server notifies master before and after split starts and needs to be updated. META. Table so that the client can know that there is a new region. Rearrange the directory structure and data files in HDFs. Split is a complex operation. The current execution status is logged when the split region is executed, and is rolled back according to the state when an error occurs. Represents the process that is executed in Split. (The red line represents the operation of region server or master, and the Green Line represents the operation of the client.) ）

1.region server decided to split region, the first step, region server was created in zookeeper in

Under the/hbase/region-in-transition/region-name directory, create a znode with a status of splitting.

2. Because Master has the region-in-transition Znode to do the monitoring, so, Mater know that the parent region needs split

3.region Server creates a subdirectory named ". Splits" in the directory of the parent region of HDFs

4.region server shuts down the parent region. Forces flush caching and marks the region as a downline state in the local data structure. If the client just requests to the parent region at this time, Notservingregionexception will be thrown. The client then makes a compensatory retry.

5.region Server creates directories and the necessary data structures in the. Split directory for two daughter region respectively. Then create two references to files that point to the parent regions.

6.region server in HDFs, create a real region directory, and move the reference file to the corresponding directory.

7.region server sends a put request to the. Meta. Table, and set the parent region to a downline state in the. Meta. Table, and two daughter region information in the row corresponding to the parent region. But at this time. META. Daughter region in the table is not yet a standalone row. This time if client scan. META. Table, you will find that the parent region is split, but the client does not see the information for daughter region. When this put succeeds, the parent region split is being executed. If the region server fails before RPC succeeds, master and the region server that next opens the parent region will clear the dirty state of this split. But when RPC returns results to parent region, that is. META. After a successful update, the process for region split will continue. Equivalent to a compensation mechanism, the next time you open this parent region will be the appropriate cleanup operation.

8.region Server opens two daughter region accept write operations.

9.region server adds information about daughters A and B in the. META. Table, after which the client can discover the two new regions and send the request to the two new region. The client is locally specific. META. Table cache, when they visit the parent region, they find that the parent region is offline and will be re-accessed. META. Table gets the latest information and updates the local cache.

The status of the 10.region server update Znode is split. Master will know that the status is updated, and master's balance mechanism will determine if the daughter regions need to be assigned to another region server.

11. After split, Meta and HDFs will still have references to the parent region. When the compact operation occurs in daughter regions, the data file is rewritten, and the reference is gradually removed. The garbage collection task periodically detects daughter regions if there is a reference to the parent files, and if no reference is directed to the parent files, the parent region is deleted.

Reference connection:

Http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/Hbase Split

http://hbase.apache.org/book/regions.arch.html HBase Official document (region)

Http://blog.javachen.com/2014/01/16/hbase-region-split-policy/split strategy

http://blackproof.iteye.com/blog/2037159 Split Source parsing

Three ways of Hbase split and the split process