Troubleshooting and solution for notservingregionexception in hbase Cluster

Author: those things |ArticleCan be reproduced. Please mark the original source and author information in the form of a hyperlink

Web: http://www.cnblogs.com/panfeng412/archive/2012/11/04/hbase-how-to-resolve-not-serving-region-exception.html

During the reading and writing process, the hbase cluster may temporarily deprecate the region due to region split or region blance. When the client and hbase cluster perform RPC operations, a notservingregionexception exception will be thrown, as a result, the read/write operation fails. Based on the actual project experience, we will describe in detail the process of discovering and troubleshooting this problem.

1. Problems Found

During stress testing on hbase clusters, it is found that when the actual amount of data written to and queried from hbase is several times that of normal times (the cluster size is 10 ~ 20, the amount of read and write data per second is tens of thousands of records), resulting in a certain degree of fluctuations in the read and write of the cluster. The details are as follows:

1) The write end throws the following exception information:

Org. Apache. hadoop. hbase. Client. retriesexhaustedwithdetailsexception: Failed 150 actions: notservingregionexception: 150 times, servers with issues: my161208.cm6: 60020, At org. Apache. hadoop. hbase. Client. hconnectionmanager $ hconnectionimplementation. processbatchcallback (hconnectionmanager. Java: 1600) At org. Apache. hadoop. hbase. Client. hconnectionmanager $ hconnectionimplementation. processbatch (hconnectionmanager. Java: 1376) At org. Apache. hadoop. hbase. Client. htable. flushcommit (htable. Java: 916)

2) The read end also throws similar exception information:

Org. Apache. hadoop. hbase. Client. retriesexhaustedexception: failed after attempts = 10, exceptions: Mon Oct 29 14:03:09 CST 2012, org. apache. hadoop. hbase. client. scannercallable @ 3740fb20, org. apache. hadoop. hbase. notservingregionexception: Org. apache. hadoop. hbase. notservingregionexception: xxxxxx, \ x0fp \ x8d \ xc3 \ xdb1053223266: \ x00 \ x00v6, 1351490475989. bd68113129f07163dc25e78fba17ad6c. is closing

The preceding exceptions occur cyclically during the stress test. Therefore, hbase clusters are unavailable for a short period of time.

2. troubleshoot

By viewing the hbase master running log and when the client throws an exception, it is found that the shard in the hbase cluster is being split and the region balance between different machines, why is the above process triggered periodically? It also occurs during the stress test (the data volume is several times larger than usual ). The following is an analysis based on the table design:

1) because the rowkey In the table has a time field, you need to create a new region every day. In addition, due to the large amount of data written, the region split operation of hbase is further triggered, this process generally takes a long time (from the perspective of online logs during testing, the average value is about 10 seconds, and the region size is 4 GB), and the region split operation is triggered more frequently;

2) at the same time, the region distribution is uneven due to the region split operation, which triggers hbase to automatically perform the region balance operation. During the region migration process, the region will also be deprecated, this process takes a long time (from the online logs during testing, the average time is about 20 seconds ).

3. Solve the Problem

First of all, from the perspective of the client, it is necessary to ensure that the read/write requests can continue after the cluster recovers when the region is offline and unavailable. The following measures can be taken:

1) For the write end, you can add records that have not been successfully written to a client cache, and submit the records to a background thread for resubmission after a period of time; you can also use setautoflush (flase, false) to ensure that records that fail to be submitted are not discarded. Stay in the client writebuffer and wait until the next time the writebuffer is full and try again until the submission is successful.

2) For the read end, after an exception is caught, retry after hibernation for a period of time.

3) Of course, you can adjust the configuration options of hbase. Client. retries. Number and hbase. Client. Pause according to the actual situation.

Then, from the server perspective, you need to solve the region split and region balance respectively:

1) During table creation, we have considered the even distribution of data on different region servers, and created and allocated the same number of region servers in advance, therefore, in order for the cluster to provide stable services in the actual online environment, you can choose to disable the auto-Balance Function of hbase, of course, after turning it off, you can trigger a balance operation when the daily read and write pressure is low (such as after the early morning.

2) Next, region is always created and cannot be reused. How can this problem be solved? The root cause is that the rowkey contains the timestamp field, and the timestamp always increases up every moment. However, the user must be able to perform Sequential Scan operations based on the timestamp field. Therefore, the timestamp field must be retained. Here we provide two solutions:

• A common method is to split a table by time, for example, by day. In this way, you can create a region partition by creating a table in advance to avoid frequent triggering of region split and other processes in the actual read/write process, however, the disadvantage of this method is that the table needs to be created in advance every day, and this DDL process may cause read and write problems. At the same time, the read and write end needs to be adapted across days, to read and write the newly created table.
• In fact, we can change the table's rowkey structure to change the timestamp field to a periodic cycle timestamp, such as the value after timestamp % ts_mode, ts_mode must be greater than or equal to the TTL time period of the table to ensure that data is not overwritten. After this transformation, region can be reused to avoid the infinite rise of region. There are also minor changes to the read/write end. During read/write operations, you only need to modulo the timestamp field and use it as the rowkey for read/write. In addition, the read end must consider adapting to the processing during scan [starttsmode, endtsmode] and [endtsmode, starttsmode.

4. Summary

The above is a summary of my problems encountered in the actual project, for your reference only. Welcome to the discussion.