HBase regularly deletes records

APIS involved in this article:

Hadoop/HDFS: http://hadoop.apache.org/common/docs/current/api/

HBase: http://hbase.apache.org/apidocs/index.html? Overview-summary.html

Begin!

How does HBase regularly and quantitatively delete records?

Use Case Analysis: some of the records we store in HBase may be fast-growing and do not need to be permanently stored, such as a large number of "system logs ", you may only need to save records for the last few months. Our storage space is very limited, especially the multi-copy disaster recovery storage such as HDFS. In addition, when HBase stores each row of data, it needs to save a rowKey for each column. If a row contains 10 columns, it will store 10 copies of the Light rowKey. The overhead can be imagined. Therefore, regular quantitative deletion has become a common requirement.

1. How to delete data regularly?

Use table-level properties: TTL (Time To Live) To set the record validity period. After the current Time exceeds the record validity period, the record will be automatically deleted. The record validity period = TimeStamp + TTL;

2. How to delete data when the data exceeds the threshold?

For example, if a table occupies a maximum of 1 TB of space, the oldest part of the table is deleted when the data exceeds 1 TB.

1. At the HDFS level, obtain the space occupied by the table.

Create a FileSystem instance through the Configuration instance, call getContentSummary (Path f) of Fs to obtain the ContentSummary instance of the table directory, and then call getLength () to obtain the table size.

2. If the table size exceeds the threshold, delete a certain number of records with a small timestamp.

Use the setTimeRange method of scan in hbase api to filter the data to be deleted, and then delete it ~