Some technical practice of Druid in Xiaomi Company

Introduction: Druid As an open-source real-time Big data analysis software, since its inception, by virtue of their own outstanding characteristics, not only gradually in the technology circle harvest more and more popularity and reputation, and has become a lot of technical team solutions in a key link, so that really in many companies in the technology stack won a place.
This article through to the Xiaomi company technical team to Druid's practice case and the experience introduction, lets everybody has the more comprehensive and the thorough understanding to the Druid, hoped can help you to study druid this young technology with more effort.
This article is selected from "Druid Real-time Big data analysis principle and practice".

Xiaomi Company was formally established in April 2010, is a focus on high-end smart phones, internet TV and smart home ecological chain construction of innovative technology enterprises.
"Let everyone enjoy the joy of science and Technology" is the vision of Xiaomi company. Xiaomi Company uses the Internet model to develop products, with the spirit of the Craftsman to do products, with the Internet mode to save the intermediate links, is committed to enabling everyone around the world to enjoy high-quality technology products from China.
　　　　　　　　　　　
Xiaomi Big cloud Platform technology architecture

DRUID enables real-time collection of large amounts of event data in a data analysis layer, rapid business analysis, and application in multiple scenarios. This paper introduces some technical practices of Druid in Xiaomi statistical products and Xiaomi advertising platform.

Scenario One: Xiaomi statistical services

Xiaomi Statistics is a mobile app data statistics service provided by Xiaomi for app developers, helping developers to understand application development, channel promotion effect and user participation information, so that developers can better optimize the experience and operation, and promote the continuous development of products. The entrance to Xiaomi Statistics is tongji.xiaomi.com and the service interface is as follows.
　　　　　　　　　　　　　　　
Xiaomi Statistical Service Interface

Real-time data analysis of important needs, in the product development process, also experienced several technical stages, these phases are not completely mutually exclusive, but applied to different scenarios and time.
The first phase: data is stored in Hadoop and analyzed and processed through a mapreduce script. A part of a complex task is executed in days, and the result is eventually written to an RDBMS like MySQL.
The second stage: in the course of business development, MySQL quickly became a bottleneck, there are two reasons, one is the database schema change cost is high, the new business needs to add new columns and new tables, the process is cumbersome and need to design the schema, second, in the case of large write operations, The increased load on the database can result in degraded read performance and occasional deadlocks. In order to solve these problems, HBase is introduced as the main storage database, and the column family of HBase is used to increase the data column conveniently. In addition, HBase is also more usable than MySQL.
The third stage: in order to improve the real-time data, the late increase of the storm distributed computing model, the use of storm can easily carry out a variety of complex data processing, a variety of aggregation and processing needs to be implemented through the program, add a data dimension, the change is relatively large, need to change from upstream to downstream overall. The advantages of this method are good reliability, high data processing ability, and can be optimized in various angles.
Stage four: Many of the data queries of millet statistics are to select some indexes and filter conditions, many scenes are similar to traditional data warehouses, so introduce Druid to deal with some standard reports of real-time data query scenarios. The data flow goes through kaa and tranquility in turn, and finally into the Druid cluster. The Druid cluster will ultimately provide the most recent day of data query functionality and allow direct user access.
　　　　　　　　　　　　　　　　　
Xiaomi statistical data stream

Druid, as a real-time analysis database, enhances the real-time data analysis capabilities of the Xiaomi Big data platform and commercial products division.

Scenario Two: Real-time data analysis of advertising platform

Druid from the advertising business, Xiaomi advertising platform also uses Druid for real-time data analysis, to help real-time analysis of various dimensions of the line changes, including on-line deployment of real-time monitoring analysis, A/B testing of the effect of the query, some fine-grained data analysis.
There are two paths for advertising data processing: One is real-time data flow, through Druid processing, mainly for internal real-time data analysis needs, and the other is through the Mini-batch way.
The DataSource (data source) of the data includes:

• Xiaomi Advertising trading Platform (Xiaomi Ad Exchange, MAX): The Scheduling management platform for AD traffic.
• Advertising Platform Billing Analysis module: Advertiser billing, various dimensions of data.
• Advertising Media analysis data: the request, presentation and other data of each advertisement media.

For example, the Advertising billing Analysis module, DRUID will include real-time advertiser billing information, which is used for internal data analysis, not for the Advertiser platform. The Advertiser launch platform uses Mini-batch mode to update and aggregate data results in a replay-based manner.
In the process of using Druid, you will encounter some problems.

1. About the query interface

Druid Query language is not particularly friendly, in the first phase after the deployment of DRUID, we developed a set of Druid query interface, mainly to meet the needs of the business, the initial effect is good, but with the increase of data sources, each time to increase the data source will need to develop some additional interface, increase the dimension, There is also a need to modify the front-end engineering, so the efficiency is not high. In the later work, the pivot tool was tried, the function is easy to use, and gradually replaced the custom query interface.

2. About query efficiency

Druid Most of the time performance is good, but if the long-range query, the system will become very slow. To solve this problem, the data sources that are frequently queried can be divided into two parts: one is a data source aggregated at a minute level, the data is kept for 10 days, and the other is a data source aggregated at the hour level, with data maintained for 2 years. Every night, aggregate hour-level data to avoid high-load cluster time. The relationship between aggregate granularity and query efficiency is as follows.
　　　　　　　　　　　　
The relationship between aggregation granularity and query efficiency

3. Deployment situation

The Druid cluster handles nearly tens of billions of event requests per day, with clusters of nearly 10 machines, the number of indexing services and historical nodes equal, and the number of machines increasing as the number of events increases. When the data source is rapidly increasing at a certain time, the system index file occupies a high CPU, which sometimes affects the normal query performance.
In the first phase, we tried to use traffic control at the service level, but later gave up. The reason is that the data expires after 1 hours, so if there is data that cannot be entered into the system, the data may be lost. As a result, we try to get the data into the Druid system, although it occasionally brings system peak pressure.
The Druid-based architecture and data flow are as follows.
　　　　　　　　　　　　　
Druid-based architecture and data flow

"The paper came to the end of shallow, know this matter to preach", as learning other technologies, the best way to Master Druid is practice, so we have a certain understanding of Druid should be as soon as possible after the practice, and strive to apply it to their own practical work, in the fight to learn fighting, Let the problems encountered in practice drive their learning and understanding of Druid technology.

This article is selected from the "Druid Real-time Big data analysis principle and practice", click this link can be viewed in the blog post view of the book.
　　　　　　　　　　　　　　　　　　　　
Want to get more good articles in time, you can search "blog point of View" or scan the QR code below and follow.
　　　　　　　　　　　　　　　　　　　　　　　

Some technical practice of Druid in Xiaomi Company