Hbase (3)-framework structure and process

 

 

 

 

 

Hbase relies on hadoop's HDFS as the storage base. Therefore, the structure is similar to hadoop's master-slave mode. hbase master server is responsible for managing all the hregion servers, however, the hbase master server does not store any data in hbase. The table in hbase logic is defined as a region stored on an hregion server. the correspondence between the hregion server and region is one-to-many. Each hregion is physically divided into three parts: hmemcache, hlog, and hstore, representing the cache, log, and persistence layers respectively. Let's take a look at the functions of these three parts through an update process:

 

 

 

 

As can be seen from the process, the update submission operation will be written into two parts of the entity. hmemcache and hlog are used to create a cache in the memory to Improve the efficiency, this ensures that some recently operated data can be quickly read and modified. hlog is used as the transaction log for synchronizing hmemcache and hstore. When the hregion server periodically initiates the flush cache command, the data in hmemcache will be persisted to hstore, and the data in hmemecache will be cleared. Here we use a simple policy for data caching and synchronization, for details about the complexity, refer to Java's garbage collection mechanism.

When reading the region information, read the content in hmemcache first. If it is not obtained, read the data in hstore again.

Several details:

1. each flash cache generates an hstore file. More and more files are stored in the hstore, which also affects the performance, therefore, when the threshold value of the set file quantity is reached, the files merge will be a large file.

2. When setting the cache size and flush interval, you need to consider the memory consumption and performance impact.

3. Each time the hregion server restarts, it will load the data in hlog that is not flushed to hstore to hmemcache again. Therefore, the high speed of hmemcache may also have a direct impact.

4. The B-tree algorithm is used to store data in hstore file. Therefore, quick locating and acquisition of column and family data operations are also supported.

5. hregion can be split by merge or by hregion, depending on the size of hregion. However, hregion will be locked and unavailable during these operations.

6. hbase master server uses meta-Info table to obtain information about the hregion server and region. A region at the top of Meta is a virtual root region, you can use root region to find the actual region below.

7. the client obtains the region server where the region is located through the hbase master server, and then interacts directly with the region server. The region server does not communicate with each other and only interacts with the hbase master server, it is monitored and managed by the master server.

 

 

 

Http://wiki.apache.org/hadoop/Hbase/HbaseArchitectureArchitecture and implementation

 

There are three major components of the hbase architecture:

1. The H! Basemaster (analogous to the bigtable master server)

2. The H! Regionserver (analogous to the bigtable tablet server)

3. The hbase client, defined by org. Apache. hadoop. hbase. Client. htable

Each will be discussed in the following sewing.

 

Hbasemaster

 

The H! Basemaster is responsible for assigning regions to H! Regionservers. the first region to be assigned isRoot RegionWhich locates all the meta regions to be assigned. EachMeta Region
Maps a number of user regions which comprise the multiple tables that
Particle hbase instance serves. Once all the meta regions have been
Assigned, the master will then assign user regions to the H! Regionservers, attempting to balance the number of regions served by each H! Regionserver.

It also holds a pointer to the H! Regionserver that is hosting the root region.

The H! Basemaster also monitors the health of each H! Regionserver, and if it detects a H! Regionserver is no longer reachable, it will split the H! Regionserver's write-ahead log so that there is now one write-ahead log for each region that the H! Regionserver was serving. After it has accomplished this, it will reassign the regions that were being served by the unreachable H! Regionserver.

In addition, the H! Basemaster
Is also responsible for handling table administrative functions such
On/Off-lining of tables, changes to the table Schema (adding and
Removing column families), etc.

Unlike bigtable, currently, when the H! Basemaster
Dies, the cluster will shut down. In bigtable, A tabletserver can still
Serve tablets after its connection to the master has died. We tie them
Together, because we do not currently use an external lock-Management
System like bigtable. The bigtable master allocates tablets and a lock
Manager (Chubby) Guarantees atomic access by tabletservers to tablets. hbase uses just a single central point for all H! Regionservers to access: the H! Basemaster.

 

The meta table

 

The meta table stores information about every user region in hbase which has des a h! Regioninfo
Object ining information such as the start and end row keys,
Whether the region is on-line or off-line, etc. And the address of
H! Regionserver that is currently serving the region. The Meta table can grow as the number of user regions grows.

 

The root table

 

The root table is confined to a single region and maps all the regions in the meta table. Like the meta table, it contains a H! Regioninfo object for each meta region and the location of the H! Regionserver that is serving that meta region.

Each
Row in the root and meta tables is approximately 1kb in size. At
Default region size of 256 MB, this means that the root region can map
2.6x105 meta regions, which in turn map a total 6.9x1010 user regions, meaning that approximately 1.8x1019 (264) bytes of user data.

 

Hregionserver

 

The H! Regionserver is responsible for handling client read and write requests. It communicates with the H! Basemaster
To get a list of regions to serve and to tell the master that it is
Alive. region assignments and other instructions from the master "Piggy
Back "on the heart beat messages.

 

Write requests

 

When a write request is wrongly Ed, it is first written to a write-ahead log calledHlog.
All write requests for every region the region server is serving are
Written to the same log. Once the request has been written to the hlog,
It is stored in an in-memory cache calledMemcache. There is one memcache for each hstore.

 

Read requests

 

Reads are handled by first checking the memcache and if the requested data is not found, the mapfiles are searched for results.

 

Cache Flushes

 

When
The memcache reaches a retriable size, it is flushed to disk,
Creating a New mapfile and a marker is written to the hlog, so that
When it is replayed, log entries before the last flush can be skipped.
A flush may also be triggered to relieve memory pressure on the Region
Server.

Cache
Flushes happen concurrently with the region server processing read and
Write requests. Just before the new mapfile is moved into place, reads
And writes are suincluded until the mapfile has been added to the list
Of active mapfiles for the hstore.

 

Compactions

 

When
The number of mapfiles exceeds a retriable threshold, a minor
Compaction is performed med which when lidates the most recently written
Mapfiles. A major compaction is saved med periodically which
When lidates all the mapfiles into a single mapfile. The reason for not
Always merge Ming a major compaction is that the oldest mapfile can be
Quite large and reading and merging it with the latest mapfiles, which
Are much smaller, can be very time consuming due to the amount of I/O
Involved in reading merging and writing the contents of the largest
Mapfile.

Compactions
Happen concurrently with the region server processing read and write
Requests. Just before the new mapfile is moved into place, reads and
Writes are suincluded until the mapfile has been added to the list
Active mapfiles for the hstore and the mapfiles that were merged
Create the new mapfile have been removed.

 

Region splits

 

When
The aggregate size of the mapfiles for an hstore reaches a retriable
Size (currently 256 MB), a region split is requested. region splits
Divide the row range of the parent region in half and happen very
Quickly because the child regions read from the parent's mapfile.

The
Parent region is taken off-line, the region server records the new
Child regions in the meta region and the master is informed that
Split has taken place so that it can assign the children to Region
Servers. shocould the split message be lost, the master will discover
Split has occurred since it periodically scans the meta regions
Unassigned regions.

Once
The parent region is closed, read and write requests for the region are
Suincluded. The client has a mechanic for detecting a region split and
Will wait and retry the request when the new children are on-line.

When
A compaction is triggered in a child, the data from the parent is
Copied to the child. When both children have saved med a compaction,
The parent region is garbage collected.

 

Hbase Client

 

The hbase client is responsible for finding H! Regionservers that are serving the specified row range of interest. On Instantiation, The hbase client communicates with the H! Basemaster to find the location of the root region. This is the only communication between the client and the master.

Once
The root region is located, the client contacts that region server and
Scans the root region to find the meta region that will contain in
Location of the user region that contains the desired row range. It
Then contacts the region server that is serving that meta region and
Scans that meta region to determine the location of the user region.

After locating the user region, the client contacts the region server serving that region and issues the read or write request.

This information is cached in the client so that subsequent requests need not go through this process.

Shocould
A region be reassigned either by the master for load balancing or
Because a region server has died, the client will rescan the meta table
To determine the new location of the user region. If the meta Region
Has been reassigned, the client will rescan the root region
Determine the new location of the Meta region. If the root region has
Been reassigned, the client will contact the master to determine
New root region location and will locate the user region by repeating
The original process described above.