FAQs about hbase [updating continuously]

This article summarizes common problems encountered during hbase development and usage, hoping to answer some questions about newly added hbaser.

1. Is the htable thread secure?

Htable is NOT thread-safe. We recommend that you use an htable object in a thread. Do not share htable objects between different threads.

In addition, to improve client efficiency, different htable objects share the same configuration object and hbase server metadata information. For details, see the htable section in the hbase authoritative guide.

2. How to Use htablepool?

Before hbase 0.90, The htable object was obtained by using the gettable method in htablepool, And the puttable method was called to store the htable object back to the pool.

In Versions later than hbase 0.90, The puttable method is not called after use, but the htable close method is called to store the htable object back to the pool.

3. What is the difference between the caching of ingress and batch?

Caching is the number of records retrieved from the server at a time, while batch is the number of keyvalues returned to the client at a time (in a result object.

For example, if each record has five keyvalues and only sets caching = 10, if the number of matched records in the table in the scan table process is greater than 10, next () the operation returns 10 results from the server. Each result contains 5 keyvalues:

*****

*****

*****

*****

*****

*****

*****

*****

*****

*****

If batch = 2 is set at the same time, 17 keyvalues will be returned for one next () operation:

**

**

*

**

**

*

**

**

*

**

4. Is put (list <put> puts) more efficient?

Put (list) is used to increase throughput. First, it is parallel between different region servers. On the same region server, the same region is parallel, while on different region servers, It is serial.

Because on the same region server, if put (list) is executed on a region, it is packaged to write a log, which is equivalent to parallel writing. If it is executed on different region servers, it is a for loop, rather than going through the thread pool, so it is serialized.

5. Is increment correct and reliable?

At present, the increment cannot be completely correct. It may be incorrect when the timeout and region server failover occurs on the network. Therefore, you need to carefully select the appropriate one based on the Application scenario.

As for the performance of increment, the performance is good, especially in hbase 0.94 and general put write operations, which can reach more than 7000 of each region server.

6. Is the number of region as much as possible?

Excessive region affects the flush efficiency and region server failover efficiency. In addition, the stress on HDFS is relatively high; If region is too small, it affects reading, because the number of storefiles is large, if the number of storefiles per region is small (online clusters are generally maintained at 1. X ~ 2. x storefiles.

The number of region writes has little impact, unless region is extremely small and the write pressure is extremely high. For example, a region server has only one region.

At present, hbase version 0.90 recommends that each region server have a region of less than 3000, while hbase 0.94 recommends a region of less than 1000, even if there are hundreds of region servers.

7. What is the overhead of Wal operation logs?

Generally, when you put data to the region server, If you enable Wal log writing, the server may need 1 ms ~ 2 ms overhead. If you do not enable Wal logs, the server only has about 0. XMS overhead, basically all of which are memstore memory operations.

In addition, in either case, the network transmission delay RTT overhead (generally between MS and MS) is required for a put operation ~ Within 1 ms ).

8. How high is the Write Performance of a single form region?

Without any optimization, Java APIs can reach about 800 TPS per second when Wal is enabled, and 1000 + TPS when Wal is disabled, which is basically the maximum network speed; thrift interfaces include about 10 ~ 20% performance loss. The actual test shows that the TPS can reach about 550 per second in the network RTT = 0.7ms environment. In the network RTT = 350 ms, the TPS can only reach about.

The above write performance data is far from the actual hbase write performance because optimization is not considered, in practical applications, you can use region, multi-client, multi-thread, batch write, and other methods to limit the performance of each region server.

Note:In the case of single form region, if there is no problem in low-concurrency writing in a short period of time, and if there is a high-concurrency writing, the request may be blocked due to region split.

9. How high is the Write Performance of a single region server?

To stress test the write capability of a single region server, you can create a new test table and create multiple region (for example, 10 ~ And then generate rowkey and value randomly and write them to the hbase cluster for stress testing. The final test result is divided by the number of region servers in the hbase cluster, you can get the write performance value of a single region server.

According to the test results obtained from the hbase cluster testing team, for the region server with 16-core CPU and 24 GB memory (the datanode process allocates 1 GB of memory and the hbase regionserver process allocates 16 GB of memory), if Wal is not enabled, A single region server can reach more than 3 million TPS. If Wal is enabled, a single region server can reach 9kb ~ 2 w tps. Hbase's thrift interface can reach native Java API 80% ~ Performance of about 90%.

10. What is the problem with null region caused by timestamp in rowkey?

Assume that rowkey contains a timestamp and TTL is set to automatically delete expired data. As the timestamp value increases, expired data will be deleted, but empty region still exists, too many region will affect the flush efficiency and region server failover efficiency. In addition, the pressure on HDFS is relatively high. At the same time, too many blank region will also occupy some metadata information.

There are two methods to solve this problem:

1) create different tables by time period: for example, create different tables by day and switch tables to read/write by day. The disadvantage of this method is that you need to create tables every day, since DDL transactions are performed by the master, as long as there is a problem with the master, it is easy to affect the transaction. In general, to prevent the impact of the master, you need to build the table a few days in advance.

2) process the timestamp field: the current timestamp is not directly stored in rowkey, instead, it stores the timestamp values that have been accumulated since 00:00:00 on Monday of the week (in seconds or milliseconds). In this way, the table can store a maximum of region values for one week, the previous region will be reused after the start of next week to avoid the expansion of region. Correspondingly, the query end also needs to process the response to the timestamp. Considering the scalability, if the data to be saved exceeds one week, this method can be extended from one week to one month.

{{Updating continuously}}