Opentsdb is an open source database based on HBase storage time series data, which is exactly the application of hbase, and its processing of time series data can be used for reference by other systems. This article will explore and discuss the design of the database.this text link: http://blog.csdn.net/bluishglc/article/details/31052749, reprint please indicate the source!
This article is based on the first stable version of Opentsdb 1.0.0, after the download deployment is complete, we need to understand the first is its database schema, it has two tables: Tsdb-uid and Tsdb. The former describes the metadata associated with the indicator (metrics), which stores the time series data. First, let's take a look at "indicator "(metrics) concept, a simple indicator is a need to collect data items, but only the indicator is not a comprehensive description of the data generated by the relevant background information, such as: if we want to count CPU usage, we can set up a name called
proc.stat.cpuMetrics, if we collect a lot of CPU information from different machines and users, if we don't identify a piece of information, we can't tell which data comes from which machine, so we also need to create some "tags" to identify a piece of data. Strictly speaking, there is no necessary dependency between the indicator and the tag, just as the data of two different indicators may have the same host tag as the one that indicates which hosts it comes from, but one thing is certain:
for a piece of data, there should be at least one indicator and one label , This kind of data is meaningful, so in Opentsdb's table design,"Indicator "(metrics) andthe tag has been uniformly placed in thestored in the Tsdb-uid table,format is:
RowKey (
self-increment id,3 byte array): NAME:METRICS,NAME:TAGK,NAME:TAGV
, and the
inverse correlation Between them is also stored.
In fact, we see that for data, the indicator to the data is a one-to-many parent-child relationship, the label on the data is a one-to-many parent-child relationship, Opentsdb
here is very typical of the design, in fact, this is the HBase table design of a common "Pattern" :
Expand the relationship between the tables, and store the data for Rowkey with the result of join! Including forward correlation and inverse Association two kinds of data! (Please refer to Figure 1 carefully)
below Let'sInsert2 x metrics:
proc.stat.cpuand the
Proc.stat.mem,and a record:
proc.stat.cpu 1297574486 54.2 Host=foo type=userto look at the data table structure
:
First,
tsdb-uid table:
Figure A
The records from the table indicate:
1. First record: Rowkey is \x00, with 3 fields: METRICS,TAGK,TAGV, whose values are the number of all indicators, label names, and tag values that have been added. This piece of data is generated and maintained by the system. There are two metrics:cpu and Mem, two key:host and type, two value:foo and user, so the value of three data Rowkey for \x00 is 2
2. A UID is allocated for a combination of one indicator + one label name + one tag value, i.e.:An
indicator + a label name + a Label value = a uid, that is to say: a uid corresponding to thean indicator + a label name + a combination of tag values is the smallest unit that can be extracted separately from the rows of statistics!
And then we look at
TsdbTable:
Figure II
Let's look at the rowkey of the record sheet:
Indicator UID (indicator + tag combination) + Data generation time (take hour time) + label 1-key uid+ label 1-vlaue uid+...+Label NThe uid of the uid+ tag N-vlaue of the key
Let us take the chart as an example, focusing on the processing of time:
1297574486 = 2011-02-13 13:21:26
MWEP = 01001101 01010111 01100101 01010000 = 1297573200 = 2011-02-13 13:00:00 (intercept hourly hour)
PK 0101000001101011 1286 (seconds deviation from hour to record time, 1286 seconds is 21分钟26秒)
1297573200+1286=1297574486
PK, that is, the number of seconds in the hour is treated as column
Some design tips:
1. Coping Strategies for Hot spot
Opentsdb deals with typical time-serialized data, and inevitably faces a "hot spot" issue, which is specifically addressed in the official document Http://hbase.apache.org/book/rowkey.design.html of HBase regarding its handling of hot issues:
However, the difference is, the timestamp are not in the
LeadPosition of the key, and the design assumption is that there be dozens or hundreds (or more) of different metric types. Thus, even with a continual stream of input data with a mix of metric types, the Puts is distributed across various point S of regions in the table.
In general, if you use time to do rowkey, you must precede the "hash" field (that is, salted processing). But Opentsdb does not have a particular hash field, and it handles smarter: First, the Time field is not placed at the beginning of the Rowkey, and secondly, the Rowkey start position picks up an ideal business field "metrics" to replace the "hash" field.
From the processing of OPENTSDB, we can conclude that, when processing time series data, if there is an "ideal" "natural" hash field in the system, it should give precedence to its starting part as Rowkey, followed by the Time field, but if such a field is not found then set the manual hash field
2. Rowkey's design ideas
One. In order to be able to retrieve the data point of a specific Metrics,tag Name,tag name, it is obvious that the Metrics,tag Name,tag name is programmed into Rowkey, but there are two obvious problems with directly using them to compose Rowkey : 1. It takes up a lot of storage space (because these values can be repeated in a lot of Rowkey) 2. Since the length of each Metrics,tag Key,tag value is not fixed, it is not conducive to directly locating them by byte offsets. (You will need to use a specific delimiter, and to avoid parsing errors due to the possibility of a specific delimiter in the input information, and to escape the delimiters for all input information)
Around a performance indicator, there will be a number of additional "attributes" (or "tags") to explain and describe it, then the query for the indicator is naturally expanded with these labels or tag values, so the rowkey of an indicator record must include these labels and tag values. But since labels and label values are variable, This brings trouble to the design of the Rowkey, so you need to assign a fixed-length ID to these tags and tag values, use their IDs in Rowkey to refer to them, so that the rowkey can be normalized to easily intercept the required "parts" from the Rowkey directly through the offsets. Two. Tall-narrow and Wide-flat two kinds of table design style combination
