HBase System Architecture

HBase system Architecture diagram

　　

Constituent Parts Description
Client:
Communicating with Hmaster and hregionserver using the hbase RPC mechanism
Client communicates with Hmaster to manage class operations
Client and Hregionserver data read and write class operation
Zookeeper:
Zookeeper Quorum storage-root-table address, Hmaster address
Hregionserver the Ephedral way to zookeeper, Hmaster feel the health of each hregionserver at any time
Zookeeper avoid hmaster single point problem
Hmaster:
Hmaster there is no single point of issue, HBase can start multiple hmaster, through the zookeeper Master election mechanism to ensure that there is always a master running
Mainly responsible for the management of table and region:
1 Manage users to change the table and delete the operation
2 Manage Hregionserver load Balancing, adjust region distribution
3 Region split, responsible for the distribution of the new region
4 after Hregionserver outage, responsible for failure hregionserver on region migration
Hregionserver:
The most core module in HBase, primarily responsible for responding to user I/O requests and reading and writing data to the HDFs file system

　　

Hregionserver manages some column hregion objects;  
Each hregion corresponds to a region in the table, Hregion consists of multiple hstore,  
Each hstore corresponds to the storage of one Column family in the table;  
Column family is a centralized storage unit. It is more efficient to put column with the same IO characteristics in a column family

Hstore:
The core of hbase storage. Made up of Memstore and StoreFile.
Memstore is sorted Memory Buffer. The process by which a user writes data:

　　

client write-on to Memstore, until Memstore full, flush into a storefile, until a certain threshold is increased to Trigger Compact Merge Operations-multiple StoreFile merged into one storefile, with version merging and data deletion, when the Storefiles compact is gradually becoming more and more large storefile- When a single storefile size exceeds a certain threshold, the split operation is triggered, and the current region is split into 2 region,region to be rolled down, The new split of the 2 children region will be hmaster assigned to the corresponding hregionserver, so that the original 1 region of the pressure can be diverted to 2 region.
As a result of this process, hbase only adds data, and the resulting update and delete operations are done in the compact phase, so the user writes only need to go into memory to return immediately, thus ensuring I/O is high performance.

HLog  
introduced HLog reason:  
in a distributed system environment, there is no way to avoid system errors or outages, once hregionserver exits unexpectedly, Memstore memory data will be lost, the introduction of Hlog is to prevent this situation  
working mechanism:  
There is a Hregionserver object in each Hlog, Hlog is a class that implements the write Ahead log , each time a user action is written to Memstore, a copy of the data to the Hlog file is written, the Hlog file periodically scrolls out of the new file and deletes the old one (data that has persisted to the storefile). When the hregionserver unexpected termination, hmaster through zookeeper sense, hmaster first processing the legacy hlog files, the different region of the log data split, respectively, placed in the corresponding region directory, Then redistribute the failed region, pick up the hregionserver of these region in the process of load region, will find that there is a history hlog need to deal with, so will replay Hlog data into Memstore, Then flush to Storefiles to complete the data recovery.

hbase storage format  
All data files in HBase are stored on the Hadoop HDFs file system in two main formats:  
1 hfile HBase keyvalue Data storage format, hfile is a hadoop binary format file, in fact StoreFile is the hfile do lightweight packaging, that is storefile the bottom is hfile 
2 HLog file, The storage format for the Wal (Write Ahead Log) in HBase, which is physically sequence File

for Hadoop

hfile

　　

picture explanation:   The
hfile file is not long and the length of the fixed block is only two: trailer and fileinfo 
The pointer in trailer points to the starting point of the other data block  
file info Records Some meta-information about files, such as: Avg_key_len, Avg_value_len, Last_key, COMPARATOR, Max_ Seq_id_key et   the
Data index and Meta index block record the starting point of each data block and Meta Block  
The data block is the basic unit of HBase I/O and, for efficiency, Hregionserver is based on the LRU block cache mechanism  
the size of each data block can be specified by parameters when creating a table, the large block facilitates sequential scan, and the small block is useful for random queries  
each data block in addition to the beginning of the magic is a keyvalue stitching, magic content is some random numbers, to prevent data corruption

Each keyvalue pair inside the hfile is a simple byte array. This byte array contains many items and has a fixed structure.

　　

Keylength and Valuelength: Two fixed lengths, respectively, representing the length of key and value  
Key part: Row length is a fixed-length value , which represents the length of the Rowkey, Row is rowkey 
Column Family length is a fixed-length value that represents the length of Family  
and then Column Family, Then there is the qualifier, then the two fixed-length values representing the time stamp and key Type (Put/delete)   The
Value section has no such complex structure, which is purely binary data

HLog File

　　

The Hlog file is an ordinary Hadoop Sequence file,sequence The key is the Hlogkey object, the Hlogkey records the attribution information written to the data, in addition to table and region names, but also includes Sequence number and Timestamp,timestamp are "write Time", the starting value of sequence is 0, or the last time the file system was deposited in sequence.
The value of HLog sequece file is the KeyValue object of HBase, which corresponds to KeyValue in hfile

HBase System Architecture