HBase Go client Row construction considerations

This is a creation in Article, where the information may have evolved or changed.

• 1. Hbase's Go Client language usage
• 2. The row usage considerations for HBase
  • 2.1. Row the first few fields to hash as far as possible
  • 2.2. Row is sorted by a dictionary of all the characters in the row

We recently used hbase in a project to do the storage of log data, on top of it to do some data analysis work, relative to Java, team members of the use of Go more skilled, so naturally use go as the client's development language, has never dealt with hbase before, Originally a relatively simple task, Leng stumbled to do a long time ...

This article only describes the considerations for the row construction of HBase

1 How to use the Go client language for Hbase

HBase officially does not have a go client, but it provides the thrift service, and we can use the go language to develop a thrift client to operate on HBase by sending RPC requests to the thrift server in HBase. A simple request flow is as follows:

Go client–-> hbase Thrift server–-> hbase

The HBase Thrift server is provided by HBase, and they also provide Thrift service description file, as for how to use Thrift service description file to claim the code of Go client, there are many tools on github.com. Thrift officials also have tools.

2 The row usage considerations for HBase

2.1 The first few fields of row to hash as far as possible

HBase is a cluster service that will distribute data across the backend storage machine based on row, and if your raw key has obvious aggregation, the row corresponding data will be concentrated on a few back-end machines, so that when the data volume is particularly large, the pressure of reading and writing is concentrated on these machines. Affects your performance, the first recommendation is to hash the raw key so that the raw key with obvious aggregation is distributed evenly (or approximately evenly) to the different back-end storage machines that are most commonly used to make md5sum

The sort of 2.2 row is to sort all the characters in the row in a dictionary

This is very important, in the afternoon fell into this pit half a day to climb out. To illustrate this problem, simplify our scenario by simplifying the following example: Writing hbase is inserting a bunch of time-stamped data, and reading HBase is reading data between the start and end of the time period. Our row has two keywords: a string data and a timestamp timestamp, based on these two keywords we construct the row method is: substring (md5sum (data), 0, 8) + data + timestamp row in the first part of the say , is the string, the key is this timestamp how to construct? At first I was using the following method:

BUF: = Make ([]byte, + len (data) IP: = []byte (pack.data) tmp: = MD5. Sum (data) copy (Buf[0:8], tmp[0:8]) copy (Buf[8:8 + len (data), data) binary. Putvarint (Buf[8+len (data):], timestamp)

　　After writing to HBase in this way (for example, I write data=aabbcc, timestamp=123456 data), reading data sometimes in start=0, end=234567 can read the data, in Start=1, end=234567 But can't read the data, baffled. Later found that the original is written timestamp error, please see the following example:

BUF: = Make ([]byte, 8) binary. Putvarint (buf, Int64 (1423484126)) fmt. PRINTLN (BUF) binary. Putvarint (BUF, Int64 (2)) fmt. Println (BUF)

　　Its output is:

[188 147 197 205 10 0 0 0] [4 147 197 205 10 0 0 0]

　　So although timestamp 1,423,484,126:2, but after the construction of row, if the data is the same, then 2 of the dictionary order is ranked 1423484126 ratio behind. This will produce the strange phenomenon mentioned above. Fortunately, go does not disappoint us, it provides a convenient way to complete the things we want to do, see the following example:

BUF: = Make ([]byte, 8) binary. Bigendian.putuint64 (buf, UInt64 (1423484126)) fmt. PRINTLN (BUF) binary. Bigendian.putuint64 (BUF, UInt64 (2)) fmt. Println (BUF)

　　Its output is:

[0 0 0 0 84 216 164 222] [0 0 0 0 0 0 0 2]

　　OK, problem solved. After exposure to unfamiliar knowledge, or to first understand the basic principles, although the time will be more investment, but from the overall income, but it is a better way.

Author:cobbliu

Created:2015-02-10 Tue 01:03

Emacs 24.4.1 (ORG mode 8.2.10)