Troubleshooting of a slow query request in hbase

Author: those things |ArticleCan be reproduced. Please mark the original source and author information in the form of a hyperlink

Web: http://www.cnblogs.com/panfeng412/archive/2013/06/08/hbase-slow-query-troubleshooting.html

Recently, the hbase cluster encountered a slow query request. The following describes the problem and the troubleshooting process.

1. Problems Found

There is an hbase table in the project. After the early morning, a batch of data will be imported in batches. The imported data volume is large, in the order of tens of millions to hundreds of millions, and then the query service will be provided to users in the day. One day, it was suddenly found that when the table only scanned a few pieces of data according to each region (256 in total), the response time of some region query requests was very slow, up to 10 seconds or even dozens of seconds.

2. troubleshoot

First, check the monitoring interface of the region server that comes with hbase. We can see that each region in this table has only 1 ~ Three storefiles exclude slow query response due to too many storefiles.

Check that the table TTL is 5 days, so a large amount of expired data exists. At the same time, because this table imports a batch of data every morning (more than 3.22 records are imported in total on the first day of last week), and the major compact cycle configuration of the cluster is 7 days, even though the data on 3.22 has expired so far, it has not been triggered by major compact to clear expired data. Therefore, a large amount of expired data has not been cleared yet, as a result, even if only a few pieces of data are scanned by each region, a large number of expired data needs to be filtered out. (from monitoring, we can see that the block cache access volume is about twice as high as usual, as shown in) to scan the actual useful data, so the query response time is very slow.

3. Solve the Problem

There are two solutions to this problem:

1) after data is imported every morning, a major compact operation is triggered (see the majorcompct method of hbaseadmin for asynchronous execution), so that the expired data in each region in the table can be cleared in time.

2) because the cluster's major compact cycle is 7 days, and the table's TTL is 5 days, you can reduce the major compact cycle (set the parameter to hbase. hregion. majorcompaction, in milliseconds; at the same time, hbase. offpeak. start. hour can be set to the hour when major compact is started. For example, if it is set to 1, it can be triggered after.) at the cluster level, major compact can be triggered as early as possible.