HBase Rowkey Design __hbase

Building a schema

The Hbase mode can be established or updated through the Hbase shell tool or by using the Admin class in the Hbase Java API.

The HBase table must be in a disabled state when the column family changes. For example:

Configuration config = hbaseconfiguration.create ();
Admin admin = new Admin (conf);
String table = "myTable";

Admin.disabletable (table);

Hcolumndescriptor cf1 = ...;
Admin.addcolumn (table, CF1);      Add new Column Family
hcolumndescriptor cf2 = ...;
Admin.modifycolumn (table, CF2);  Modify the Column family

admin.enabletable (table);

Number of column families

HBase the current treatment of two or 3 row families is not very good, so we should try our best to keep the number of the columns small. At present, the flushing and compactions operations are based on each region, so if most of the data of a column family is flush, it will cause the neighboring column family to flush, even if its data amount is very small. When there are flush and compaction operations in many column families, a large number of I/O requests are caused. (Can be resolved by changing the flush and compaction operations based on the column clan.) ）

See compaction for more information compaction.

Use only one column family as much as possible, introducing a 2nd, 3-column family when and only if your data access is at the column level. For example, one of your data accesses will only request data from the same column family, but will not cross the list of family requests. cardinality of the column family

When a table has multiple column families, you should be aware of the problem of cardinality (for example, number of rows). If column family A has 1 million rows and column family B has 1 billion rows, then the data of column family A is likely to spread to many region, which will make the scan operation efficiency of the column family a lower. Rowkey Design Hot Spots

The rows in the HBase are sorted in a Rowkey dictionary order, which optimizes the scan operation and allows the associated rows and rows to be read together to be accessible in a nearby location for scan. However, poor rowkey design is the source of hot spots. Hotspots occur in a large number of clients directly accessing a cluster of one or very few nodes. Access can be read, write, or other operation. A large number of accesses can cause a single machine with hotspot region to exceed its own affordability, causing performance degradation or even region. This also affects other regions of the same regionserver, because the host cannot serve other region requests. Design a good data access model to enable the cluster to be fully and evenly utilized.

To avoid writing hotspots, design rowkey make different rows in the same region, but in more data, the data should be written to multiple region of the cluster rather than one. Here are some common ways to avoid hotspots and their pros and cons: adding salt

The addition of salt is not the addition of the rowkey in cryptography, but the addition of random numbers to the front. Specifically, assign a random prefix to Rowkey to make it different from the previous sort. The number of prefix types assigned should be the same as the number of region you want to scatter the data to. If you have some hot rowkey repeatedly appearing in other evenly distributed rwokey, adding salt is useful. Consider the following example: it disperses write requests to multiple regionservers, but it has some negative effects on reading. Add Salt Example

If you have the following rowkey, each of the region in your table corresponds to each letter in the alphabet. With the beginning of ' a ' is the same region, ' B ' begins with the same region. In the table, all that start with ' F ' are in the same region, their rowkey like this:

foo0001
foo0002
foo0003
foo0004

Now, if you need to disperse this region to 4 region. You can use 4 different salts: ' A ', ' B ', ' C ', ' d '. Under this scheme, each letter prefix will be in a different region. After adding salt, you have the following rowkey:

a-foo0003
b-foo0001
c-foo0004
d-foo0002

So, you can write to 4 different region, theoretically, if everyone writes to the same region, you will have 4 times times the throughput.

Now, if you add another row, it will randomly assign one of the a,b,c,d as the prefix and end with an existing row as a tail:

a-foo0003
b-foo0001
c-foo0003
c-foo0004
d-foo0002

Because the allocation is random, you need to do more work if you want to retrieve the data in a dictionary order. Adding salt this way increases the throughput at write time, but there are additional costs when reading. Hashes

In addition to adding salt, you can also use hashes, which make the same line always add salt to the same prefix. Hashing can also spread the load across the cluster, but reading can be predictable. Using a determined hash allows the client to refactor the completed Rowkey, using a get operation to obtain a normal fetch of a row of data. Hash Example

Like the example given in the salt-adding method, you can use some sort of hashing method to make the prefix of a rowkey such as foo0003 always ' a ', and then, in order to get a row, you can get the corresponding rowkey by hashing. You can also optimize the hashing method so that certain rowkey are always in the same region. Flip Key

A third way to prevent hot spots is to flip a fixed-length or number-formatted rowkey. This allows the often-changed part of the Rowkey (the most insignificant part) to precede it. This can be effective random rowkey, but at the expense of the Rowkey order. monotonically increasing rwokey (continuous sequence of time)

In the Hadoop Authority guide, there is an optimized point of attention: when all clients are consistently writing to a region for a period of time, then write to the next region. For example: This behavior can occur as a monotonically increasing rowkey (timestamp). You can view the Kai LAN comics monotonically increasing values are bad. Explains why monotonically increasing rowkey is problematic in distributed table Systems (Hbase). This monotonically increasing rowkey accumulation in the same region problem can be mitigated by randomization of input records. But generally we should avoid using timestamps or sequences (1,2,3) as primary keys.

If you do need to store time series data in HBase, you can learn opentsdb, which is a successful example. A link schema has a page that describes the patterns used by Opentsdb in HBase. The key pattern in OPENTSDB is: [Metadata type] [timestamp], at first glance this seems to violate the principle of not using timestamp as Rowkey, however, the difference is that the timestamp is not in the critical position of Rowkey, and the design assumes that there are many metadata types. Therefore, even if there is continuous input data mixed with meta data, they are put into different regions in the table. try to reduce the size of rows and columns

In HBase, value is always transmitted with its key. When a specific value is transmitted between systems, its rowkey, column name, and timestamp are also transmitted together. If your rowkey and column names are large and can even be compared to a specific value, you will experience some interesting situations. The index in the HBase storefiles (which helps with random access) eventually occupies a large amount of memory allocated by HBase, because the value and the key are large. You can increase the block size so that the Storefiles index increases at greater intervals, or modify the table's schema to reduce the size of Rowkey and column names. Compression also helps with larger indexes.

Most of the time small inefficiencies are irrelevant, but in this case, any access mode requires the column family name, column name, Rowkey, so they are accessed billions of times in your data. Row Family

Make the column family name as short as possible, preferably one character. (For example: ' d ' stands for data/default). Property

A lengthy property name ("Myveryimportantattribute") is readable, but a shorter property name ("via") is better stored in the hbase. Rowkey Length

It is reasonable to make the rowkey as short as possible, which is useful for the necessary data access (Get,scan). But when the short key is useless for data access, it has a better get/scan attribute than the long key. When designing Rowkey, we need to weigh and compromise. byte mode

Long shaping is 8 bytes and you can store unsigned numbers to 18,446,744,073,709,551,615 with 8 bytes. If you store a character as a string, you need nearly 3 times times the byte.

You don't believe me? Here is the sample code you can run directly:

Long
//
long L = 1234567890L;
byte[] lb = bytes.tobytes (l);
System.out.println ("Long bytes Length:" + lb.length);   Returns 8

String s = string.valueof (l);
byte[] sb = Bytes.tobytes (s);
System.out.println ("long As String length:" + sb.length);    Returns

//hash
//
messagedigest MD = messagedigest.getinstance ("MD5");
Byte[] Digest = Md.digest (Bytes.tobytes (s));
SYSTEM.OUT.PRINTLN ("MD5 Digest bytes Length:" + digest.length);    Returns

string sdigest = new string (digest);
byte[] Sbdigest = bytes.tobytes (sdigest);
SYSTEM.OUT.PRINTLN ("MD5 Digest As String length:" + sbdigest.length);    Returns 26

Unfortunately, using a binary type makes your data difficult to read outside of your code, for example, when you add a value, the following is what you see in the shell:

HBase (Main):001:0> incr ' t ', ' r ', ' F:q ', 1
COUNTER VALUE = 1

hbase (main):002:0> get ' t ', ' R '
column
  
   cell
 f:q                                          timestamp=1369163040570, value=\x00\x00\x00\x00\x00\x00\x00\x01
1 row (s) in 0.0310 seconds

  

The shell will try to print the string, but in this case it can only print 16. This will also happen in your rowkey. If you know what you're saving, that's fine, but if any data is stored in a specific value, it's hard to read. This also needs to be weighed. Flip Time Stamp

A common database processing problem is to get the most recent version of the data, and using a rollover timestamp as part of the Rowkey is useful for this issue, and you can append Long.max_value-timestamp to the end of the key, for example: [Key][reverse_ Timestamp

The most recent value of [key] in the table can get the first record of [key] by scan [key], because the Rowkey in HBase is ordered, the newest [key] is up to any older [key], so the first record is the newest one.

This technique can replace the use of multiple versions of data, and multiple versions of data will permanently (for a long time) save all versions of the data. At the same time, this technique uses a scan operation to obtain all versions of the data. Rowkey and the row family

Rowkey is visible to the column family, so the same rowkey can exist for every column family in the table without conflicts. invariance of Rowkey

Rowkey can not be changed, the only way to change is to delete this line, and then insert this line. This is quite a common problem, so the first time the right to get Rowkey is worthwhile, the relationship between Rowkey and region split

If you want to prescaler your table and understand how your rowkey is important to region boundary distribution, consider this example, use the 16 character that can be displayed as key position. (For example "0000000000000000" to "FFFFFFFFFFFFFFFF") run these key sequences on 10 region through Bytes.split (this is a split strategy when creating region through Admin.createtable (byte[] startkey, byte[] EndKey, numregions mode) will produce the following result:

48 48 48 48 48 48 48 48 48 48 48 48 48 48 48 48//0 54-10-10-10-10-10-10-10-10-1                 0-10-10-10-10-10-10//6 61-67-67-67-67-67-67-67-67-67-67-67-67-67-67-68  = 68-124-124-124-124-124-124-124-124-124-124-124-124-124-124-126//D 75 75 75 75 75 75 75 75 75 75                                A/K 82 18 18 18 18 18 18 18 18 18 18 18 18 18 18 14 R 88-40-40-40-40-40-40-40-40-40-40-40-40-40-40-44//X 95-97-97-97-97-9 7-