Design and implementation of hbase large object storage scheme

Design and implementation of hbase large object storage scheme

Nanjing University Kang Yi

The era of massive data is coming, with the rapid development of the Internet, network access, network access logs, communications records, video data, mobile networks and a variety of intelligent terminals produced by the huge data set is also expanding dramatically. An important feature of the dataset is that more than 80% of the data is unstructured. Traditional technology can not be qualified for the analysis, management and mining of large data sets, and the industry is hbase for a popular solution to the large-scale processing. Unlike a generic relational database, HBase is a database suitable for unstructured data storage. and unstructured data as a large object (SCM object), hbase to its processing and other structured data, so, in the HBase data import process, due to a large number of unstructured data import, HBase region size increased rapidly, The split process of the region and the compact process will frequently promote, to a certain extent, stuck to the client's writing, affecting the HBase insert performance. Thus, if the split and compact number of HBase region can be reduced during insertion, the insertion performance of hbase can be greatly improved. At the same time we also need to take into account the performance of their reading and storage management complexity, and without introducing external factors while minimizing the modification of HBase source code. Based on these factors, it presents its own hbase large object storage (SCM object Storage, LOB) solution.

Design and implementation of hbase large object storage scheme