HBase Learning notes a---basic concepts

In front, this article is part of the content from the Community official website after a moderate translation, part of the experience summary, part of the Plagiarism Network blog, (do not enumerate the references, in this apology) together, the purpose of this article, hope to summarize some useful, should pay attention to the things, the basic idea is to put forward a topic, in this topic , the related things added in, rather than separate chapters separately introduced, although the reason is poor, but conducive to the late search by keywords.
      Apache HBase is the short name of Hadoop database, and HBase is a distributed, extensible, data-oriented, large-scale datastore.      hbase Basic concepts      1, region             region are part of the data, so is a self of all data, but the region includes a complete line, so region is a subset of the Behavior Unit table.            Each region has three main elements: which table it belongs to, the first row it contains (no first line in the first region) the last line he contains (no last line in the last region) &NB Sp           When the table is initially written, the table has only one region, and as the data grows, the region begins to grow larger, and when it reaches the defined threshold size, the change Split to two size basically the same region,       and this threshold is storefile set size (parameter: hbase.hregion.max.filesize new version default 10G)   at the first split region, all loaded data is placed on that server in the original area, and as the table becomes larger   the number of   regions increases correspondingly, the region is the smallest unit of the HBase cluster distribution data.            (but region is also made up of block, specifically the block and HDFs block what kind of relationship, the region is a single regionserver, This part of the region may be transferred to other machines unless the regionserver is down, or otherwise hangs, or when the balance is executed. )           * This is why the region is less, causing the region to distribute uneven, always assigned to a few nodes, read and write concurrency effect is not significant, this is HBase The reason why the reading efficiency is lower.       2, lock           hbase Lock is a row lock, regardless of the number of things accessed on the row, the update for this row is atomic, either succeeds or fails, There is no case of partial success. This means that if you update only one column in a row of 1000 columns      , it also locks the positive row.       3, hbase metadata table           hbase internal maintenance of two metadata tables, respectively-root-and. META. Table their respective maintainers the list, status, and location of all region of the current cluster. The-root-table contains the. META. Table's region list, because. META.       tables may be split over the size of region, so-root-will save. META. Table's Region index,-root-table is not split. and. META. The table contains a list of all user region       User-space. The items in the table use the region name as the key. The region name is MD5 hash value by the name of the owning table, the starting line of region, the time created, and the whole.            For example:                  &NBSP ; award_week_2013,201311:170100626,1371697714559.f7f37b98c01f68b7b5cc6c1c3734a666                  |--table name-------------|-----start line---------|--Create timestamp---|----Overall MD5 hash value------------|            This reminds us of table SCIf the StartRow and Endrow properties are added to the process of an, the lookup is very fast, because it is based on the given Rowkey. META. Table   Locate the region, and then find it from      region.            When the value of this table will change.            When region is split, disable, enable, drop, or balance cause region redistribution or because regionserver When the region is reassigned. Meta. Table data will be updated in a timely manner,       this will ensure that the tables accessed by meta are present.                   4, Access process client clients first connect to zookeeper this is where you'll find-root-first. The client then obtains the scope of the requested row by-root-. The location of the meta.region. The client then looks for it. Meta.region to get the node and location where the User-space region resides. Then the client can interact directly with the Regionserver in the region where the manager is.          Note:               Each row operation may have access to three remote nodes, in order to save these costs, The client caches the locations where they traverse-root-and. META. And the start and end lines of User-space region, so that each visit does not       will be queried from the table, but what if it changes. There is this problem, so the client will be wrong, then the region is undoubtedly moving, then the client will again from. META. Find the       new location in region and put it into the cache again and again. Same thing, if. META. The region of the mobile, the client will go to-root-table query. Meta.region's new location.            but due to hbase design problems, when HBase is used in the MapReduce process, an error occurs in the region of the access cache, and after the error, the cache updates, Task tries again to correctly pass            The following are the bugs we often encounter:           2013-09-24 0 1:26:15,487 ERROR org.apache.hadoop.hbase.regionserver.hregionserver:            Closing Scanner for    TMP_TOPLIST_GOLD_TOTAL_1001,,1379956828778.4A8E5F51EAFC48AED247913C715A8CC1.            org.apache.hadoop.hbase.notservingregionexception:region is not online: Tmp_toplist_gold_total_1001,,1379956828778.4a8e5f51eafc48aed247913c715a8cc1.            Based on this situation, it is recommended not to change the number of task attempts to 1, which can lead to many job fail            These two parameters are:  mapred.map.max.attempts Map task maximum number of attempts by default is 4  mapred.reduce.max.attempts reduce the maximum number of attempts to try the default is 4 & nbsp      5, Zookeeper's role   &NBSp          Zookeeper is simply a service that coordinates and serves distributed applications.             Zookeeper Quorum In addition to storing the address of the-root-table and Hmaster address, hregionserver Also registered to zookeeper in ephemeral way so that hmaster can feel the health of the various       Regionserver at any time, and zookeeper through the election way Avoids the single point problem of Hmaster. Ensure that at any time, there is only one master in the cluster that stores all the address entries for region. Real-time monitoring of the status of the Regionserver, the region server's on-line and offline information real-time notification to master storage HBase schema, including which table, each table has what column family                    6, Hmaster         &NBSP ;   said, zookeeper avoids the single point of the Hmaster problem, an HBase cluster can start multiple hmaster, and zookeeper through the election way to ensure that only one hmaster in the cluster in live state         Others are dormant, and zookeeper wakes up other dormant hmaster if there is a problem with hmaster.                hmaster What is the main function of the functions?                1, manage user's add to table, delete, change, check operation          &NBS p;     2, management RegionseRVer load Balancing, adjust region distribution                3, after the region split, Distribute the new region to different regionserver.                4, after regionserver outage, the region managed on Regionserver Reassigned by the Hmaster.           summarizes Hmaster's main tasks: 1, htable DDL Operations 2, Region assignment. The rest is basically done by client and Regionserver.       7, Hregionserver             Hregionserver are primarily responsible for the respective user I /O request, which in turn interacts with HDFs to read and write data from HDFs, although each process is important, the individual considers hregionserver to be the most central process in hbase.             Hregionserver A brief description of the internal structure:             Hregions erver internal management of a series of Hregion objects, hregion and region is a matter. In fact, hregion corresponds to a region,hregion in the table is the encapsulation of it. Each hregion       consists of multiple hstore.             Hstore corresponds to the column Family in the table, regardless of how much data is inside the column Family, a new hstore is created, So putting data of the same attribute into the same column Family very       necessary to avoid one visit, access to multiple HSTore, low performance. While Hstore is the core of HBase storage unit, and Hstore consists of two parts, a memstore, and then storefile              Memstore is sorted Memory Buffer, the data written by the client is first written to Memstore, and when the Memstore threshold is reached, it is flush into a storefile (hfile), StoreFile Is the             file that is stored on the hard disk, how much is this threshold.             Hbase.hregion.memstore.flush.size   This parameter represents the size of each memstore, Of course the system defaults to 134217728 bytes, which is 128M, which takes into account the number of region each RS bears.               The function of this parameter is to flush all the memstore of the region when the sum of all memstore sizes in a single range exceeds the specified value. Regionserver Flush is handled asynchronously by adding a queue to the request, simulating production       consumption patterns. There is a problem here, when the queue is too late to consume, generating a large backlog of requests, can lead to memory spikes, the worst case is to trigger oom.               It says that when all memstore in the region exceed the specified value, flush is triggered, and the previous one says that each column Family a hstore, that multiple column Family this       problem comes out, if one hstore data volume is large, and the other one hstore only a few data, will flush at the same time?   Yes, this illustrates a problem when you try to set up a single columnfamily       table when the HBase table is designed, otherwise hbase does notCan deal with the above problems very well.               two parameters in addition to the above parameters               Hbase.regio Nserver.global.memstore.upperLimit default 0.4 is 40%                 to prevent Memstore from occupying total memory Too large, when Regionserver all region reaches total heap memory 40%,hbase will block all updates, to flush all memstore, and release                Memstore-occupied memory               Hbase.regionserver.global.memstore.lowerLimit default is 0.35 35%               This parameter indicates that when the R Egionserver all Memstore up to 35% of total heap memory, triggering flush individual memory-intensive memstore, which is going to do block, write update also            is subject to impact.       8, hbase data storage           1, hbase all data is stored on the Hadoop HDFs file system, And this part of the file includes two file types:                 One is hfile hadoop binary file, actually storefile to hfile Made a lightweight wrapper, so the bottom of the storefile is hfile.                 Two is the Hlog file, HBase's Wal (Write Ahead Log), which is actually in the form of sequence file inside Hadoop. What is sequence file will be described in the following HADOP chapters, now       understand that it is a text store file consisting of a byte stream of Key/value that is serialized by binary.                  The following is the hfile's storage format, which is also an official figure.                                  then post the second picture:                                    combined with the above two graphs to see more intuitive, The first picture is the format of the hfile file, the following figure is the detailed structure of the keyvalue in the data Block, if the first graph is equivalent to a molecule, then the second one is the atom, the smallest of which cannot be divided.                 See the first: hfile file is not long, only two blocks of FileInfo and Trailer store meta information for each file , such as Avg_key_len,last_key,comparator,max_seq_id_key, data index and meta index each record the starting point of each Data block and meta block. The Data Block is the basic unit of HBase I/O, which sees that region is finally in block, but this block can be set manually. &nbSp               For example: Create ' newtable ',{name=> ' info ',blocksize=> ' 131072 '}, I just set the blocksize of family info to 132072   is the 128M default is 65536 64M just and Hadoop hdfs   block that is dfs.block.size The default is 64M is the same? error, Hbase column family data block size is 65536 bytes that is 64k is the HDFs Block size of 1/1024, this place and HDFs is a difference, with such fine granularity, the purpose of the block operation is more efficient loading and Cache data, which does not depend on the size design of the HDFs block, but only belongs to an attribute within HBase, while the HDFS block is designed to be 64M convenient for mapreduce use.                 and then open the data to see what's inside it, Magic and a lot of keyvalue.                 The magic content is random numbers to prevent data corruption, and keyvalue again zooms in on the second picture, KeyValue:                       Start is two fixed-length values that indicate the length of the key and the Long value Degree. Next is the key, which starts with a fixed-length value, representing the length of the Rowkey, followed by the Rowkey, then the fixed-length value, representing the length of the   family, then the family, then the qualifier, then the two fixed-length values, Represents the time stamp and key Type (Put/delete). The value section does not have such a complex structure, which is purely binary data.                 &NBSP    hlog file agencies do not do a detailed explanation, is sequencefile files, only do hlogfile is if the work of the introduction:                      client want HBase to write data, it will write Memstore and HLog at the same time, when the interval is certain, Memstore is permanently stored on the hard disk, That is, by Memstore flush into storefile, then this part hlog will be deleted                 But, When the Memstore not written to the hard drive, Regionserver suddenly hung up what to do? That's when the hmaster, it will hlog according to the distribution of the different region, and will hang the regionserver of the region heavy                 new assignment to new Regionserver, when Regionserver Load region, found the hlog to handle, The data recovery is done by writing to the Memstore in replay log mode.                                  &NBS P                                                                        &nbsp