Kylin Environment Construction and operation

Kylin is an open source, distributed OLAP Analysis engine that provides SQL interfaces and OLAP interfaces based on Hadoop to support terabytes-to-PB data volumes. Kylin environment before the installation, need to install Jdk,mysql,tomcat,hadoop,hbase,hive,zookeeper and other software. Because we use Ambari to manage the cluster, these work has been done, only a brief introduction of Kylin environment construction.

I. Kylin Environment CONSTRUCTION

Download apache-kylin-1.5.1-bin.tar.gz and unzip, configure the environment variables for Tomcat and Kylin. Then the main configuration of kylin.properties, such as Kylin.rest.servers. Finally kylin.sh start starts kylin,kylin.sh stop closes Kylin. Login Http://hostname:7070/kylin Access KYLIN application, username and password are admin,kylin respectively.

Description: It is necessary to note that Tomcat is already included in the apache-kylin-1.5.1-bin.tar.gz.

Some of the features of Kylin are as follows:

• Hyper-fast OLAP engine with scalability
• Providing Ansi-sql interfaces for Hadoop
• Interactive query Capabilities
• MOLAP Cube
• Seamlessly integrates with other BI tools, such as tableau, while MicroStrategy and Excel will soon be available
• Job Management and monitoring
• Compression and encoding support
• Incremental update of Cube
• Leverage HBase coprocessor for query latency
• Approximate Query Capability for distinct Count (Hyperloglog)
• Easy-to-use Web management, build, monitor, and query the cube interface
• Security capability to set ACLs at Cube/project level
• Support for LDAP integration

Two. Kylin cube setup and job monitoring

We use the official own example sample.sh to learn as follows:

1. Run ${kylin_home}/bin/sample.sh and restart KYLIN.

Analytical:

Three tables were generated under the hive default database, Kylin_cal_dt,kylin_category_groupings,kylin_sales.

2. Log in to Http://hostname:7070/kylin and select project "Learn_kylin".

3. Click Build in the Actions option in the Model page.

Note: Click on the End Date input box to select an incremental build of the cube's ending dates and submit the request.

4. Click on the Monitor page

After the request is successfully submitted, you will see a new job created. Click the Job Details button to see the details displayed on the right. As shown below:

Description: The job details provide each step of its record for tracking a job. You can dock the cursor over a step status icon to view the basic status and information. Click the icon button that appears for each step to see the details: Parameters, Log, Mrjob, eaglemonitoring.

5. Click on the new query option in the Insight page

Analytical:

Select Part_dt, SUM (price) as total_selled, COUNT (distinct seller_id) as sellers from Kylin_sales Group by Part_dt Order B Y Part_dt

Three. Kylin Cube Permission Grant

Click the +grant button to authorize. A cube has four different permissions, namely Cube Query,cube operation,cube Management,cube ADMIN. There are two types of authorization objects: User and role. Role refers to a group of users who have the same permissions.

1. Granting User rights

Analytical:

Select the User type, enter the user name of the user you want to authorize, and select the appropriate permissions. Select the user type, enter the username of the user you want to authorize, and select the appropriate permissions. Then click the Grant button to submit the request. After this operation succeeds, you will see a new table entry in the table. You can select different access permissions to modify the user permissions. Click the Revoke button to delete a user who has permissions.

2. Grant Role Permissions

Select the role type, select the set of users you want to authorize by clicking the drop-down button and select a permission. Then click the Grant button to submit the request. After this operation succeeds, you will see a new table entry in the table. You can select different access permissions to modify the group permissions. Click the Revoke button to delete a group that has permissions.

Four. How Kylin works

Kylin the OLAP engine infrastructure, including the metadata (Metadata) engine, the query engine, the job engine, and the storage engine, including the rest server in response to client requests. Apache Kylin Overview, as shown below:

• REST Server: Provides some restful interfaces, such as creating a cube, building a cube, refreshing a cube, merging cubes, and so on, Project, Table, Cube and other metadata management, user access rights, system configuration dynamic modification, and so on. In addition to this interface can be implemented SQL query, these interfaces on the one hand through the call of third-party programs, the other side is also used by the Kylin Web interface.
• Jdbc/odbc interface: Kylin provides JDBC driver, driver classname for org.apache.kylin.jdbc.Driver, prefix jdbc:kylin with URL:, The flow of queries using the JDBC interface is the same as the internal process of querying with a restful interface. This kind of interface also makes Kylin very good compatible Tebleau even Mondrian.
• Query Engine: Kylin uses an open-source calcite framework for SQL parsing, which is equivalent to the SQL engine layer.
• Routing: The module is responsible for translating the execution plan generated by the parse SQL into a cube-cached query, which can be completed in seconds or even milliseconds through the precomputed cache in HBase. There are also some operations that use querying raw data (stored on Hadoop on a hive query), which is a relatively high latency query.
• Metadata:kylin has a large number of metadata information, including the definition of cube, the definition of a star model, job information, job output information, the directory information of the dimension, and so on, metadata and cube are stored in hbase, stored in the format is a JSON string, In addition, you have the option to store metadata in the local file system.
• Cube Build Engine: This module is the basis for all modules, it is responsible for the precomputed cube creation, the process is to read the raw data through hive, and then generate htable through some mapreduce computation and then load into hbase.

Reference documents:

[1] APACHE Kylin Overview: http://kylin.apache.org/cn/

[2] Distributed large data Multidimensional Analysis (OLAP) engine Apache Kylin installation configuration and usage Example: http://lxw1234.com/archives/2016/04/643.htm

[3] Apache kylin fast data cube algorithm: Http://www.infoq.com/cn/articles/apache-kylin-algorithm

[4] The ultimate OLAP engine solution for Big data: http://www.csdn.net/article/2014-10-25/2822286

[5] Kylin's OLAP engine: http://www.mamicode.com/info-detail-1015006.html

[6] Apache Kylin The Big Data era of OLAP weapon: http://www.bitstech.net/author/huwei/

[7] Kylin's Cube model: http://www.cnblogs.com/en-heng/p/5239311.html

Kylin Environment Construction and operation