Address: http://wiki.apache.org/hadoop/Hbase/HbaseArchitecture
This article from the blog garden Jing Han http://gpcuster.cnblogs.com
There is an article about hbase that is very easy to get started.ArticleFor more information, see Understanding hbase and bigtable.
Introduction
To better understand what this article is about, we strongly recommend that you read Google's paper bigtable paper first.
Hbase is an Apache open-source project. It aims to provide a storage system running in a hadoop distributed environment similar to bigtable. Just as Google has set up bigtable in its own distributed storage system GFS, hbase is based on HDFS.
Data is logically organized into tables, rows, and columns. We can use an interface similar to iterator to traverse our row data and obtain the value of this column through the value of the row. The same row corresponds to multiple column values of different versions.
Data Model
The data model used by hbase is similar to that of bigtable. ApplicationProgramStore data rows in a tagged table. A Data row contains a sorted row key value and column value of any length.
The format of a column name is"<Family >:< label>"<Family> and <label> can be any byte arrays. The <family> set of a table is fixed. To modify the <family> Field of a table, you must use the administrator. However, you can add a new <label> at any time without declaring it in advance. Hbase stores data on disks according to <family>, so all items in a <family> should have the same read/write method.
By default, a row of data is locked each time. The atomic operation is performed when the row data is written. However, the row data can be locked and then read and write at the same time.
We can also lock Multiple rows of data at a time, but this operation requires us to actually specify it.
Conceptual Model
Conceptually, a table consists of multiple data rows, and columns in each data row do not necessarily have values. The following example is from bigtable paper.
Row key |
Time stamp |
Column "Contents :" |
Column "Anchor :" |
Column "MIME :" |
|
"Com. CNN. www" |
T9 |
"Anchor: cnnsi.com" |
"CNN" |
||
T8 |
"Anchor: My. Look. ca" |
"Cnn.com" |
|||
T6 |
"<HTML> ..." |
"Text/html" |
|||
T5 |
"<HTML> ..." |
||||
T3 |
"<HTML> ..." |
||||
Physical storage model
In terms of concept, table data is stored by sparse rows, but physically they are stored by columns. Understanding this concept is important when considering table design and program design.
Return to the previous example and follow the physical storage diagram below:
Row key |
Time stamp |
Column "Contents :" |
"Com. CNN. www" |
T6 |
"<HTML> ..." |
T5 |
"<HTML> ..." |
|
T3 |
"<HTML> ..." |
Row key |
Time stamp |
Column "Anchor :" |
| "Com. CNN. www" | T9 | "Com. CNN. www" "CNN" |
| T8 | "Anchor: My. Look. ca" "cnn.com" |
Row key |
Time stamp |
Column "MIME :" |
"Com. CNN. www" |
T6 |
"Text/html" |
As shown in the preceding figure, empty columns are not stored because they do not need column-oriented storage formats. Therefore, when the request column value is"Contents :"Will return a null value.
If the value of timestamp is not provided during data request, the latest data will return the latest value. In actual storage, all values are sorted in descending order by timestamps.
Row range: Region
For a program, a data table is composed of a series of data columns in ascending order. The data rows are in ascending order by name, and the timestamp is in descending order. Physically, a data table is split into multiple data rows (tablet in bigtable ). The region of each data row ranges from the starting row (inclusive) to the ending row (excluded ). All data rows form a data table. Different from bigtable, in bigtable, a data row region is determined by the table name and end row value, while hbase is determined by the table name and start row value.
Each column family in the region is managed by an object called hstore. Each hstore consists of one or more mapfiles (a file type in hadoop. The hstore concept is similar to Google's sstable. Once mapfiles is disabled, it is a type that cannot be modified. Mapfiles are stored in hadoop HDFS. The other differences are as follows:
Mapfiles cannot map memory at present.
Mapfiles are maintained by sparse index files, while sstable is at the end of the files.
Hbase extends mapfiles, so you can improve the search performance when using bloom filter.
Architecture and implementation
Hbase consists of the following three main components:
Hbasemaster (similar to bigtable master server)
Hregionserver (similar to bigtable tablet server)
Next, we will discuss these parts.
Hbasemaster
Hbasemaster is responsible for allocating regions to hregionserver. The first allocated region is the root region, which is used to locate all the meta regions. Each meta area is used to locate user areas in some columns. Once all the meta regions are allocated, the master node allocates the user regions to the hreginserver and maintains the load balancing for each hreginserver.
At the same time, hbasemaster also contains an hreginserver address pointing to the region containing the root.
At the same time, hbasemaster will monitor the health status of each hreginserver. If an hreginserver is unavailable, it will provide the unavailable hreginserver with the hlog and table providing services by other hreginservers.
In addition, hbasemaster is also responsible for managing data tables, such as putting a table in a valid (online) or invalid (offline) state, changing the table structure (increasing or decreasing the columnfamily), and so on.
Unlike bigtable, if the hbasemaster fails, the entire cluster is disabled. In bigtable, even if the master machine fails, tabletserver can still provide services. Bigtable uses an out-of-the-box lock management mechanism, while hbase uses the single-point access mode: All hreginservers access hbasemaster.
The meta table
The meta table stores information about all user regions and hreginserver objects. The Meta table contains the key values from the beginning to the end of each row, and whether a table is valid or invalid, the address of each hreginserver, and so on. The size of the Meta table increases as the user area increases.
The root table
The root table is restricted to one region. It specifies all meta table information. This information includes the region and hreginserver information of each meta.
The size of each row in the root table and meta table is about 1 kb. The size of each region is 256 MB, which means that the root table can load the 2.6x105 meta region and convert it to the user region which is 6.9x1010, the number of bytes that can be stored is 1.8x1019 (264 ).
Hregionserver
Hreginserver is responsible for processing user read and write operations. Hreginserver communicates with hbasemaster to obtain the data tables that require services and report the running status to hbasemaster.
Write requests
When a write request arrives, it will first be written to a write-ahead log called hlog. Hlog is cached in memory, calledMemcacheEach hstore can have only oneMemcache.
Read requests
When a read request arrives, hreginserver first searches for the data in memcache. When the data cannot be found, it searches for the data in mapfiles.
Cache Flushes
When memcache reaches the configured size, a mapfile is created and written to the disk. This reduces the memory pressure on hreginserver.
Cache Flushes often occurs during reading and writing. When a new mapfile is created, the read and write operations will be suspended. The new mapfile is created and added to the hstroe management.
Compactions
When a certain number of mapfiles exceed a configured threshold, the compression operation starts. The main task of the compression operation is to periodically merge some mapfiles into a mapfile.
During the compression operation, the read and write operations of the hreginserver will be suspended until the operation is completed.
Region splits
When the mapfiles managed by an hstore exceed the configured value (currently 256 MB), the Regional splitting operation is performed. The segmentation operation divides the original region into two new regions.
The read and write operations are suspended until the operation is completed.
Hbase Client
Hbase client is responsible for finding the hreginserver that provides the required data. In this process, hbase client first communicates with hbasemaster and finds the root region. This operation is the only communication operation between the client and the master.
Once the root region is found, the client can find the corresponding meta region by scanning the root region to locate the hreginserver that actually provides the data.
After locating the hreginserver that provides data, the client can use this hreginserver to find the required data.
This information will be cached by the client, and you do not need to proceed to this process the next request.
When a region in these regions is unavailable, the client will reverse the above process until it finds the hreginserver that actually provides the data.
Client API
See introduction to htable and hbaseadmin in javadoc.
Ingress API
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
