Hadoop Database-HBase

transferred from:http://blog.csdn.net/iAm333

1 What is HBase?

HBase, a Hadoop Database, is a highly reliable, high-performance, column-oriented, scalable distributed storage System. With HBase, you can build large, structured storage clusters on inexpensive PC servers. Its underlying file system uses HDFS, using zookeeper to manage the communication between the cluster's Hmaster and each region server, monitor the status of each region server, store the entry address for each region, and so on.

2. When is hbase used?

First of all think about the traditional relational database has what characteristics, the approximate characteristics are:

1. Supports transactional, ACID (atomicity, consistency, isolation, and durability) characteristics;
2. row-type storage;
3. SQL statements are more convenient to use;
4. Support indexes, views, etc. ;

There are several scenarios in which you might consider using hbase instead of a relational database:

1. System needs to adapt to different kinds of data formats and data sources, can not pre-strict definition of the pattern, need to deal with large-scale data;
2. Does not emphasize the relationship between the data, the data to be stored is semi-structured or unstructured;
3. The data is very sparse;
4. Want to do a better job of scaling;

For example, Google will bigtable used to store the index data of the page, index data will be good to meet the above requirements.

3. What is the difference between hive and pig?

1. HBase is low latency, unstructured, and programming-oriented, while Hive is high latency, structured, and analysis oriented;
2. Hive itself does not store and compute data, it is completely dependent on the tables in HDFs and mapreduce,hive as logical tables;
3. HBase provides a large memory hash table by organizing the memory of all the machines in the node, it needs to organize its own data structure on disk and in memory, and the table in HBase is a physical table;
4. If it is a full table scan, use Hive+hadoop, and if it is an index access, use Hbase+hadoop.
5. Hive is primarily used for static structures and for work that requires frequent analysis;
6. Pig is relatively lightweight compared to hive, and its main advantage is that it can significantly reduce the amount of code by using Hadoop Java APIs directly;
7. Both hive and pig can be used in combination with hbase, and hive and pig also provide high-level language support for HBase, making data statistics processing on hbase easy.

4. the structure of HBase

1) tables, rows, columns, and cells

Let's start with a simple summary: the most basic unit is the column, one or more columns (row), and a unique row key to determine the storage. There are many rows in a table, each column may have multiple versions, and different values are stored in each cell.

HBase lines are ordered between rows and rows, sorted by the dictionary order of row key, which is unique, occurs only once in a table, or is updated on the same row, and the row key can be any byte array. A row consists of several columns, some of which can form a column family (column family), and all columns of a column family are stored in the same underlying storage file, which is called hfile.

The column family needs to be defined when the table is created and the number should not be too large. The column family name must consist of printable characters, and you do not need to define a column when creating the table. The reference format for a column is usually family:qualifier,qualifier, or it can be any byte array. The name of the qualifier in the same column family should be unique, otherwise it is updating the same column, there is no limit on the number of columns, there can be millions of. Column values also have no type and length qualification. HBase checks the length of the row key, which should be less than 65536 by default.

A visual HBase table is as follows:

The timestamp represents the timestamp, which is specified by the system by default and the user can display the settings. Use different timestamps to differentiate between different versions. The values of the different versions of a cell are arranged in descending order of timestamps, giving precedence to the most recent value when reading. The user can specify the maximum number of versions that each value can hold, and the default maximum number of versions for the HBase-0.96 version is 1.

The access mode of HBase is as follows (table, row key, column family, column, timestamp) values. That is, the value of one column of a column family of a row key in a table is unique.

The access operation of the row data is atomic, and any number of columns can be read. Cross-row transactions and cross-table transactions are not currently supported.

Data in the same column family is compressed together, and access control disks and memory are performed at the column family level.

2) Automatic Partitioning

The basic unit of expansion and load balancing in HBase, called Region,region, is essentially a contiguous storage space sorted by row keys. If the region is too large, the system will split them dynamically, instead, merge multiple region to reduce the number of files stored.

A table starts with only one region, and when the user starts inserting data into the table, the system checks the region size to ensure that it does not exceed the configured maximum, and if it is exceeded, divides it from the middle value of the row key in part two, dividing it into two parts of roughly equal size.

Note that each region can be loaded by only one region server, and each region server can load multiple region at the same time. Shows a table that is actually a logical view of the region collection that is loaded by many region servers.

The number of region that can be loaded per server and the optimal size of each region depend on the effective processing power of a single server.

3) hbase storage format

The storage format for keyvalue data in Hfile:hbase. hfile is a binary format file for Hadoop.

The storage format for the Wal (Write-ahead-log, pre-written) file in Hlog:hbase is physically the sequence file for Hadoop.

The hfile format is as follows:

hfile files are variable in length, and the only fixed ones are file info and trailer. Trailer stores pointers to other blocks, which are written at the end of the persisted data to the file, and when written, the file becomes an immutable data store file. The data blocks stores the key-values, which can be seen as a mapfile. When block is closed, the first key is written to index, and the index file is written when the hfile is closed.

The specific format of the keyvalue is as follows:

, there are four types of keytype, namely put, Delete, DeleteColumn, and deletefamily. Rowlength is 2 bytes, row length is not fixed, Columnfamilylength is 2 bytes, columnfamily length is not fixed, columnqualifier length is not fixed, timestamp is 4 bytes, The KeyType is 1 bytes. The length of the columnqualifier is not recorded because it can be calculated from other fields.

4) WAL (pre-write log)

Region server saves the data to memory until it accumulates enough data and brushes it to disk, which avoids many small files. However, if a power outage or other failure occurs, data loss occurs when the data stored in memory is not saved to disk. Wal can solve this problem. Each update (edit) is written to the log, and only when the log writes are successful does it tell the client that the write succeeds, and the server batches the data in memory on demand.

If the server crashes, region server returns to the log, causing the server to revert to the state it was in before the server crashed. Shows the write process:

• All changes are saved to the Wal before being passed to Memstore. The whole process is this:
• The client initiates an action to modify the data, such as put. Each modification is encapsulated in a KeyValue object instance and sent out via RPC calls. These calls are sent to the region Server that contains the matching region;
• When the KeyValue instance arrives, they are assigned to manage the corresponding row Hregion instance, the data is written to the Wal, and then placed into the memstore that actually owns the record;
• When the memstore reaches a certain size or undergoes a specific time, the data is asynchronously written continuously to the file system (hfile).
• If there is a problem with the write process, Wal can guarantee that the data is not lost because the Wal log hlog is stored on HDFs. Other region servers can read the log file and play back the changes to recover the data.  

5) HBase System architecture

The HBase schema includes HBase Client, Zookeeper, Hmaster, Hregionserver, and Hstore stores several parts. Described below in one by one. A general architecture diagram is as follows:

A) HBase Client

The HBase client uses the RPC mechanism of hbase to communicate with Hmaster and Hregionserver. For management class operations (such as Build table, delete table, etc.), client and Hmaster RPC, and client and hregionserver RPC for data read-write class operation.

b) Zookeeper

A distributed, open-source distributed Application coordination service that enables distributed applications to implement synchronization services, configure maintenance and naming services, and so on. It is an open source implementation of chubby.

In addition to storing the address of the-root-table and the address of Master in Zookeeper Quorum, Regionserver registers itself in Zookeeper so that master can feel the health of each regionserver at any time.

c) Hmaster

• Manage user's increment, delete, change, check operation to table;
• Manage the load balance of hregionserver, adjust region distribution;
• After the region split, responsible for the distribution of the new region;
• After Hregionserver shutdown, responsible for the regions migration on the failed hregionserver.  

d) Hregionserver

• Mainly responsible for responding to user I/O requests, to the HDFs file system to read and write data, is the most core of hbase module;
• When the user updates the data will be assigned to the corresponding Hregion server commits the changes, these changes are written to the Memstore write cache and the server Hlog file inside. Commit () Calls are returned to the client after the operation has been written to the Hlog;
• When reading the data, the Hregion server will first access the Blockcache read cache, if the cache does not change the data, will go back to the Hstores disk search, each column family will have a Hstore collection, Each of the Hstore collections contains many hstorefile files.  

e) Special Tables

-root-table and. META. Tables are two of the more special tables: Meta. The region information for the user table is recorded. Meta. can have multiple regoin. -root-recorded the. META. Table's region information,-root-only one Region,zookeeper records the location of the-root-table. Specific as follows:

5. Why is hbase fast?

The main reason that HBase can provide real-time computing services is determined by its architecture and underlying data structure, which is determined by Lsm-tree (log-structured merge-tree) + htable (region partition) + Cache-- The client can navigate directly to the Hregion server server where the data is to be looked up, and then locate the data to match directly on a region of the server, and the data portion is cached by the cache.

As mentioned earlier, HBase saves the data in memory, the data in memory is ordered, and if the memory space is full, it is written to hfile, and the content stored in hfile is ordered. When data is written to hfile, the data in memory is discarded.

The hfile file is optimized for disk sequential reads and is stored by page. Shows the process of storing and merging multiple blocks into a disk in memory, and merging writes results in a new result block, and eventually multiple blocks are merged into larger chunks.

Many small files are generated after multiple brush writes, and the background thread merges small files into large files so that disk lookups are limited to a few data store files. HBase writes faster because it does not actually write to the file immediately, but instead writes the memory first and then asynchronously brushes it into the hfile. So in the client's view, the write speed is very fast. In addition, Random writes are converted to sequential write time, and the data write speed is stable.

The speed of reading is fast because it uses the LSM tree structure instead of the B-or + + trees. The sequential read speed of the disks is fast, but the search for tracks is much slower than it is. The storage structure of hbase causes it to require the disk seek time to be within a predictable range, and no additional seek overhead is incurred by reading any number of records that are contiguous to the rowkey being queried. For example, there are 5 storage files, then a maximum of 5 disk pathfinding is required. The relational database, even if there is an index, is unable to determine the number of disk seek. Also, the HBase read is first found in the cache (Blockcache), it takes the LRU (least recently used algorithm), if not found in the cache, will be in the memory of the Memstore, only these two places can not be found when the contents of hfile will be loaded, As mentioned above, the speed of reading hfile is also very fast, because it saves the seek cost.

6. HBase Common Operations

• List
• Create
• Put
• Scan
• Get
• Delete
• Disable
• Drop

Hadoop Database-HBase