Druid Infrastructure and applications

This article describes the infrastructure and working process of Druid, and provides a deeper understanding through an application case. Durid Introduction Druid is a sequential data analysis engine for high-performance, columnstore, distributed data storage. Second-level queries that can support "PB"-level data. Similar products are available in Kylin/clickhouse. The typical application of Druid is the cube combination query analysis under OLAP scenario. such as data drillthrough (Drill-down), roll-up (roll-up), slicing (Slice), cutting (Dice), and rotation (PIVOT). The following section of the application example is elaborated in detail. Durid Infrastructure first to understand Durid main node: 1, Broker node broker node plays the role of the history node and the query route of the real-time node. It is primarily responsible for receiving external queries, forwarding queries to the nodes where each segment data resides, and aggregating the results back. 2, historical node (historical node) historical is mainly responsible for historical data storage and query, receive coordination node data loading and deletion instructions, download segment from Deepstoage, complete data loading or deletion in ZK after the notice. Historical nodes follow the shared-nothing architecture, so there is no single point of issue between nodes. Nodes are independent of each other and the services provided are simple, and they only need to know how to load, delete, and handle immutable segment. Historical nodes can also be grouped together into different historical tiers. This will give an advantage when the cluster is large. such as the hot and cold separation of data, according to different business data separation (a certain degree of resource isolation). Of course, the historical node is the core of the overall query performance of the cluster, because historical will assume the vast majority of segment queries. 3, Coordinator node (coordination nodes) is mainly responsible for the management of data and distribution on the historical nodes. The coordination node tells the history node to load new data, unload stale data, replicate data, and move data for load balancing. Load data and drop data rules can be configured. 4, Overlord node (index service can be understood as task management node) Description: Responsible for receiving tasks, management tasks. Receive external HTTP requests (new task, Query task status, kill task, etc.), assign management tasks (when there is a new task request, overload node assigns the task to Middlemanager node to execute). 5, Middlemanager node (can be understood as the Overlord node of the work node) function Description: can start n (configurable) a peon, receive Overlord assigned task, and then give yourself peon to execute.

Picture description (max. 50 words)

The 
  query process is shown by the blue arrow, the broker node receives the query (Q1), and the query is sent to the history node and the live node (Q2,Q3), in which the real-time node is the task that is started on the MM node. This task takes care of data ingestion and provides real-time data queries. Data ingestion process See red arrow, D1 is the client production data is eventually written Kafka (this process may be in the middle of the client and Kafka, also contains a number of links, such as data transmission and cleaning), D2 and D3 process is the deployment of Tranquility-kafka services, Consumption Kafka data written to the corresponding TASK,TRANQUILITY-KAKFA when started will communicate with the Overlord node, the Overlord node assignment task to Middlemanager execution. D4 is a task-responsible segment segment that ends normally, and then writes segment data to the Deepstorage process. (Real-time task run time is segmentgranularity+windowperiod+intermediatepersistperiod). D5 is the historical node that downloads segment from Deepstorage and declares in ZK the process responsible for that segment segment query. At present Druid data ingestion process also has a more recommended way is Kafka Index Service (abbreviated KIS), interested students can refer to the official documents, Kis on the Kafka version has strong requirements. Although the overall architecture of the Druid is somewhat complex, overall stability is very good, with few cluster failures. With the problem of cluster hardware failure and data itself, the SLA basically can reach 4 9. Coordinator,overlord two nodes is a master-slave mode, which guarantees two instances per role. Broker node stateless, can be a number of instances, the front of a domain name can be (in order to ensure cache hit, it is best to configure IP hash). Historical node stateless, there is a certain amount of redundancy. Middlemanager is used as a data ingestion node, and node downtime raises the risk of data loss if the task does not have a replica configured. Of course, Kis can avoid this problem. Durid data aggregation, storage core idea Druid data storage is divided into three parts timestamp, dimensions, metrics. Among them, timestamp, metrics part is the use of LZ4 direct compression. However, the dimensions section needs to support filtering queries and grouping queries. So each dimension of the Dimensions section uses the following three data structures to transcode and store:  

A Dictionary that maps values (which is always treated as strings) to integer IDs,
For each distinct value in the Column,a bitmap This indicates which rows contain that Value,and
A List of the column ' s values,encoded using the dictionary in 1
As an example, the source data is as follows:

Picture description (max. 50 words)

The key of the dictionary table is unique, so the map key is unique and the value of column Value,map is incremented from 0 onwards. The Name column of the sample data has only two different values. So the Zhang San number is 0 and John Doe number is 1:

{
"Zhang San": 0
"John Doe": 1
}

2. Column data

To save the value of this column in each row, the value is the ID instead of the original value. Because of the map dictionary above, there is a corresponding relationship:

[0,

1,

1,

0]

3. Bitmaps-one for each unique value of the column

The key of bitmap is the key (original value) of the first step map, and value is an identity of true and False (yes | no?). equals | Not equal to? ), the value is only 0, 1, as follows:

Value= "Zhang San": [1,0,0,1]

Value= "John Doe": [0,1,1,0]

所以由上可知最坏的情况可能是随着数据量的增加，bitmap的个数也成线性增长，为数据量大小*列的个数。那么在什么情况下会导致这种线性增长？这里我们引入了一个基数(cardinality)的概念。基数=unique（dim1,dim2.....），如若dim取值均为各种爆炸性id或者随机数，则druid的预聚合将完全失去意义。所以在druid的应用场景中，基数约小，聚合效率越高。讲了dimensions怎么存储，那么metrics又是怎么聚合（roll-up）呢？这就要引入druid数据schema定义了。下一章结合应用一块看一个示例。应用示例与实践经验假设有这样一份数据，典型的商品销售数据。

Picture description (max. 50 words)

我们构造成druid中的数据schema如下：

{
"DataSources": [{
"Spec": {
"Dataschema": {
"DataSource": "Test_datasource",
"Granularityspec": {
"Segmentgranularity": "Hour",
"Querygranularity": "Minute",
"Type": "Uniform"
},
"Parser": {
' Type ': ' String ',
"Parsespec": {
"Format": "JSON",
"Timestampspec": {
"Column": "Time",
"Format": "Auto"
},
"Dimensionsspec": {
"Dimensions": ["ProductName", "City", "channel", "Action"]
}
}
},
"Metricsspec": [{
"Name": "Count",
' Type ': ' Count '
}, {
"Type": "Doublesum",
"FieldName": "Price",
"Name": "Sale"
} ]
},
"Tuningconfig": {
"Type": "Realtime",
"Windowperiod": "pxxx0m",
"Intermediatepersistperiod": "pxxx0m",
"Maxrowsinmemory": "100000"
}
},
"Properties": {
"Topicpattern": "Test_datasource",
"Task.partitions": "2",
"Task.replicants": "1"
}
} ],
"Properties": {
...
}
}
In front of the emphasis on dimensions, we look at metrics. In the example above we define only the count and the doublesum for price, then these metrics have been fixed for later analysis requirements. We see the one or two rows of red in the table above, all dim values are exactly the same, querygranularity is one minute. So at this point in 2018-06-11 12:23:00, the two rows of data are aggregated into one line, count=2,sale=0. And so on

然后我们再来看看具体的分析需求，一个钻取的例子。我们首先查看商品A昨天的点击量，select sum(count) from table where productName=‘A’ and action=‘click‘，再想看看地区=北京，渠道=web呢？是不是再加几个where就搞定了？select sum(count) from table where productName=‘A’ and city=‘北京’ and channel=‘web‘ and action=‘click’; 然后就是切片和切块，也很简单，就是几个group by。这些在druid中都能非常轻松的支持。具体使用上的经验总结：1. reindex思想。一般我们实时数据查询粒度配置的会比较小，秒级或者分钟级。那么对于一天前，三天前，一个月前的数据呢？这时候一般关注的粒度将不再那么细，所以我们一般会采取redinx的策略进行再聚合 2. 针对历史数据，可能对于某些维度将不在关心，这时候我们也可以在reindex时，将无用的维度剔除掉，可能大大减少整体数据的基数。 3. 一般数据压缩比例。这里提供一个大概的参考值。数据总基数在10W以下，每天数据量约百亿左右，druid中聚合后的索引数据与原始数据大小之比可以到1：100，甚至1：1000。 4. druid适用于常规的olap场景，能非常轻松的支撑每天百亿甚至千亿级别的数据写入。 5. 爆炸性维度数据，以及频繁update数据的需求，不适用于druid的场景。总结本文主要对druid做了入门级的基础介绍，可以给大家做olap引擎技术选型时做一个参考。以及对druid的初学者做一个大致介绍。druid是一款非常优秀的olap引擎，从性能、稳定性上来说，都是非常不错的。

Druid Infrastructure and applications