NetEase Video Cloud: hbase– storage file hfile structure analysis

NetEase Video Cloud is a PAAs video cloud service launched by NetEase, which is mainly used in online education, live show, telemedicine, enterprise collaboration and other fields. Today, NetEase video cloud technology experts to share with you: hbase– storage file hfile structure analysis.

Hfile is an organization of files for HBase to store data, referencing the Tfile implementations of BigTable sstable and Hadoop. From HBase onwards to now, Hfile has undergone three versions, of which V2 was introduced in 0.92 and V3 was introduced in 0.98. HFileV1 version of the actual use of the process found that it occupies a lot of memory, hfile V2 version is optimized for this, hfile V3 version of the basic and V2 version of the same, just add the tag array support at the cell level. In view of this, this article mainly for the V2 version of the analysis, V1 and V3 version of interested students can refer to other information.

HFILE Logical Structure

The logical structure of the hfile V2 is as follows:

The document is divided into four main sections: scanned block section,non-scanned block section,opening-time data section and trailer.

Scanned block section: As the name implies, all data blocks will be read when the sequence is scanned hfile, including the leaf Index block and the bloom block.

Non-scanned Block section: Indicates that the data will not be read when hfile sequential scan, including the meta block and intermediate level data Index blocks two parts.

Load-on-open-section: This part of the data needs to be loaded into memory when the region server for HBase is started. Includes FileInfo, Bloom filter block, data block index, and Meta block index.

Trailer: This section mainly records the basic information of hfile, the offset value of each part and the addressing information.

HFILE Physical Structure

As shown, hfile will be cut into blocks of equal size block, each block size can be created by the parameter blocksize when creating a table column cluster 65535 ' Specify, default is 64k, large block for sequential scan, small block to facilitate random query, and therefore need to weigh. and all block blocks have the same data structure, as shown on the left, HBase abstracts block blocks into a unified hfileblock. Hfileblock supports two types, one type does not support checksum, and one does not. To facilitate the explanation, choose the hfileblock internal structure which does not support checksum:

The hfileblock shown mainly consists of two parts: Blockheader and Blockdata. Where Blockheader primarily stores block metadata, Blockdata is used to store specific data. The most central field in the block metadata is the Blocktype field, which is used to indicate the type of block, and HBase defines 8 kinds of blocktype, each of which stores different data contents, and some stores user data, Some store index data, and some store meta metadata. For any type of hfileblock, there are blockheader of the same structure, but the blockdata structure is not the same. The following is a simple list of the most important blocktype, which is detailed in detail for each of the Blocktype:

Block block parsing in hfile

The file is cut into several types of blocks from the hfile level, followed by a detailed introduction to several important blocks, because the index-related block is not covered in this article, and a separate article is then written to analyze and explain it. First of all, we will introduce the trailerblock of recording hfile basic information, then introduce the actual storage block DataBlock of user data, and finally introduce the block related to Bron filter.

Trailer Block

This paper mainly records the basic information of hfile, the offset value and addressing information of each part, and the data structure in trailer memory and disk, which only shows some core fields:

The hfile will parse the trailer block and load it into memory, and then load the data in the Loadonopen area, the following steps:

1. Version information is loaded first, and the version in HBase contains both MajorVersion and MinorVersion, which determines the main version of hfile: V1, V2, or V3 The latter determines whether minor corrections, such as support for checksum, are supported on the basis of the main version determination. Different versions determine read parsing of hfile using different reader objects

2. Obtain the length of the trailer according to the version information (different version of the trailer length), and then load the entire Hfiletrailer Block according to the trailer length

3. Finally load the Load-on-open part into memory, the starting offset address is the Loadonopendataoffset field in trailer, the end offset of the load-on-open part is hfile length minus trailer length, The Load-on-open section mainly includes the root node of the index tree and the FileInfo two important modules, FileInfo is a fixed-length block, it records some meta-information of the file, for example: Avg_key_len, Avg_value_len, Last_ KEY, COMPARATOR, Max_seq_id_key, and so on; The index root node is presented in the next article.

Data Block

DataBlock is the smallest unit of data storage in HBase. DataBlock primarily stores user's keyvalue data (KeyValue is usually followed by a timestamp, not shown in the figure), and the keyvalue structure is the core of hbase storage. Each data is stored in hbase with the keyvalue structure. The keyvalue structure can be represented in memory and disk as:

Each keyvalue is composed of 4 parts, key Length,value Length,key and value respectively. Where key value and value length are two fixed-length values, and key is a complex structure, first the length of the Rowkey, then the Rowkey, then the length of the columnfamily, and then the columnfamily, Finally, timestamps and keytype (there are four types of keytype, put, Delete, DeleteColumn, and deletefamily, respectively), value is less complex, a string of pure binary data.

Bloomfilter Meta Block & Bloom block

Bloomfilter is critical for the random read performance of HBase, which can eliminate unused hfile files for Get operations and partial scan operations, reducing actual IO times and improving random read performance. Here is a brief introduction to how Bloom filter works, Bloom filter uses bit arrays to implement filtering, the initial state of the lower array each bit is 0, as shown in:

If there is a collection at this point s = {x1, x2, ... xn},bloom filter uses k independent hash functions to map each element in the collection to the range of {1,..., m} respectively. For any one element, the number that is mapped to is indexed as the corresponding bit array, and the bit is set to 1. For example, if the element x1 is mapped to the number 8 by the hash function, the 8th bit of the bit array will be set to 1. The collection s only has two elements x and y, which are mapped by 3 hash functions, mapped to (0,2,6) and (4,7,10), and the corresponding bits are set to 1:

Now if you want to determine whether another element is in this collection, only need to be mapped by these 3 hash functions, to see if there is a 0 existence of the corresponding position, if any, that this element does not exist in this set, otherwise it may exist. Indicates that Z is definitely not in the set {x, y}:

HBase each hfile has a corresponding bit array, keyvalue write hfile will go through a few hash function mapping, map the corresponding array bit to 1,get request to come in after the hash map, if there is 0 on the corresponding array bit, Indicates that the data for the GET request query is not in the hfile.

The bit array in hfile is the value stored in the above bloom block, and it can be imagined that the larger the hfile file, the greater the keyvalue value stored inside it, and the larger the bit array will be. Once too large, it is not appropriate to load directly into memory, so hfile V2 is designed to split the bit array, splitting it into separate bit arrays (split by Key, and part of a contiguous key using a bit array). Such a hfile will contain a multi-bit array, according to the key query, first will be positioned to a specific array of bits, only need to load this bit array into memory to filter, reduce memory expenditure.

In the structure of each bit array corresponds to a bloom Block in the hfile, in order to facilitate the key to locate the specific need to load which bit array, hfile V2 also designed the corresponding index bloom index Block, corresponding memory and logical structure diagram as follows:

Bloom INDEX Block Structure totalbytesize represents the bit array of bits, Numchunks represents the number of Bloom block, Hashcount represents the number of hash functions, Hashtype represents the type of hash function, Totalkeycount represents the number of keys that Bloom filter currently contains, Totalmaxkeys represents the number of keys currently contained by Bloom Filter, Bloom Index entry corresponds to each Bloom filter The index entry for the block stores the corresponding bit array as an index pointing to the Bloom block,bloom block of the ' Scanned Block section '.

The structure of the Bloom Index entry is shown on the left, Blockoffset represents the offset of Bloom block in hfile, and Firstkey represents the first key corresponding to Bloomblock. According to the above, a GET request comes in, first, according to the key in all the index entries in the binary search, find the corresponding Bloom index Entry, you can locate the key corresponding to the bit array, loaded into memory for filtering judgment.

Summarize

This essay first explains the logical structure and physical storage structure of hfile from the macroscopic level, and decomposes hfile into various types of blocks logically, then trailer block and Data from microscopic perspective respectively. Block is parsed on the structure: by parsing the trailer block, you can get the hfile version and the offset of several other parts of the hfile, which can be loaded directly at the time of reading, and the data Block parsing can know how the user data is actually stored in HDFs, and finally, the storage structure of bloom filter in hfile is understood by introducing the working principle of bloom filter and the related block block. Next, we will write an article to analyze the structure of index block in hfile and the corresponding index mechanism.

Bloom Filter HBase hfile

If you are interested in our article, you can continue to focus on our NetEase Video cloud website (http://vcloud.163.com/) or NetEase video cloud official (vcloud163) to Exchange Oh!

NetEase Video Cloud: hbase– storage file hfile structure analysis