Druid.io Series 2: Concepts and architecture

Before introducing the Druid architecture, we will first understand some of the basic concepts in Druid with the basic principles of OLAP. 1 Data

In Figure 3.1, for example, combined with the basic concept of OLAP that we introduced in the first chapter, the data by column can be divided into the following three types:

Time series (Timestamp), Druid is both an in-memory database and a time-series database, and all queries in Druid and the indexing process are closely related to the time dimension. The Druid bottom uses an absolute number of milliseconds to save the timestamp, and the default is to use the ISO-8601 format to display the time (shape: yyyy-mm-ddthh:mm:sss. Sssz, where "Z" stands for 0 time zones, China is located in the East eight area can be expressed as +08:00).

Dimension Columns (Dimensions), the dimension concept of Druid is consistent with OLAP, character type (String) data in one record can be thought of as a dimension column, and dimension columns are used to filter filter, group data. As shown in Figure 3.1, page, Username, Gender, city, four columns.

measure column (Metrics), the Druid metric concept is also consistent with OLAP, where the value (Numeric) type data in a record can be thought of as a measure column, and the measure column is used for aggregation (aggregation) and calculation (computation) operations. As shown in Figure 3.1, characters Added, characters removed these two columns. 2 rolled up

In a production environment, hundreds of millions of raw data are entered into Druid each day, druid the smallest granularity supports millisecond-level events, but in general usage scenarios we rarely focus on such fine-grained data sets, and Aggregating data according to certain rules can not only save storage space, but also get a more valuable view. As with other OLAP products, Druid also supports roll-up (roll-up) operations. The most common roll-up operation is to aggregate the time dimension, such as the data in Figure 3.2 according to the hourly granularity of the aggregation can be obtained in Figure 3.3, figure 3.3 is more intuitive than the figure 3.2来 said, it is more useful for analysts to master the overall situation. However, the roll-up operation also leads to the loss of information, because the granularity of the roll-up becomes the smallest data visualization granularity, that is, the millisecond level of the raw data, if roll-up by minute granularity, then we can see the minimum granularity of the data after storage is the minute level.
3 shards

Druid is a time series database, and there is also the concept of sharding (sharding). Druid Shards the original data according to the time dimension, and each shard is called a segment (Segment).
Segment is the most basic data storage unit in Druid, which uses a column (columnar) to store all the dimension values, measures, time dimensions, and indexes that correspond to some data source (DataSource) in a certain time interval (interval).

Segment Data structure

The Time dimension (absolute number of milliseconds) and the measure are compressed using an integer or floating-point (floating-dot) array at the bottom, using the LZ4 compression algorithm (optional LZF, uncompressed) by default.

Dimension columns use dictionary encoding, bitmap indexing, and corresponding compression algorithms, including the following three data structures, with examples from Figure 3.1:

Why use these three types of data structures, and what are their advantages:

Using dictionary encoding can reduce the storage space of string data, and the expression is more convenient and compact.

The bitmap index, which is similar to the inverted index, enables the bitwise logic to operate quickly;

Bitmap index size = column cardinality * Number of data rows, for high cardinality columns, we also introduced in the second chapter a lot of bitmap index compression algorithm, druid implementation of Concisebitmap compression and roaring bitmap compression, Concise is used by default.

Segment storage Structure
The segment logical name is shaped like "datasource_intervalstart_intervalend_version_partitionnum":

DataSource: Data source;

Intervalstart, Intervalend: The beginning and end of the time interval, using the ISO-8601 format;

Version: The release number, the default V1, is used to distinguish multiple loads of segment corresponding to the same data;

PartitionNumber: partition number, within each interval, depending on the size of the data, there may be multiple partitions within a segment, Official recommendation by controlling the granularity of the time interval or the number of partition to guarantee the size of each partition between 300MB-700MB, the optimal load and query performance can be obtained.

4 cluster nodes

The Druid cluster contains a variety of node types, namely historical node, Coordinator node, Broker node, indexing Service node (including Overlord, Middlemanager and Peon) and realtime Node (including Firehose and plumber).

Druid the entire cluster into the above roles, there are two purposes: first, dividing historical node and realtime node is to cut the load of historical data and the real-time stream data processing, because both need to occupy a lot of memory and CPU; Divide coordinator node and broker node, cut the demand of the query and how the data is distributed within the cluster, ensure that the user's query request does not affect the distribution of the data in the cluster, so that the data is not "hot and cold", local overheating, affecting the query performance problems.

Figure 3.5 shows the real-time/batch data flow within the Druid cluster and the query request process. We can see that the real-time data arrives at Realtime node, through the indexing Service, the data in the time window stays in realtime node memory, and the data outside the time window is organized into segment storage into deep storage Bulk data through Indexing Service will also be organized into segment storage into deep storage, while segment meta information will be registered to the Meta Repository, coordinator nodes will periodically (default 1 minutes) to synchronize the meta database , perceiving the newly generated segment, and notifying the online historical node to load Segment,zookeeper will also update the entire cluster internal data distribution topology map.

When a user needs to query for information, the request is submitted to Broker Node,broker node requesting zookeeper to get a map of the data distribution in the cluster to know which historical node and realtime node the request should be sent to, Summarizes the return data for each node and returns the final result to the user.
In (c), we will introduce each kind of node.