kylin2.3 version enables JDBC data sources (you can generate hive tables directly from SQL, eliminating the hassle of manually conducting data to hive and building hive tables)

kylin2.3 version enables JDBC data sources (you can generate hive tables directly from SQL, eliminating the hassle of manually conducting data to hive and building hive tables)

Description

The JDBC data source, which is essentially a hive data source.

Performance is still not good because of the database Big Table Association. So the default data source for Kylin is still hive, which I think is very reasonable.

It is a convenient way to correspond to the JDBC data source. The rationale is to select the table (or SQL query) you want to use by connecting to the database.

The data is extracted in parallel by Sqoop and the corresponding hive table is generated according to the table name. The cube's build is based on the generated hive table.

Each time the build is re-extracted, the hive table is generated, and the hive table is deleted when the build is complete.

The equivalent is the tedious duplication of work that was done before the development of data that needs to be synchronized to HDFs, new hive tables, and synchronizing hive tables to Kylin.

Disadvantages:

1, based on the above description, it is easy to come to its disadvantage is that these hive tables are instantaneous. Each build will be on-site to extract the full amount of data (thereby increasing the pressure on the database, increasing the network overhead, and slowing down the overall build speed of cube). (This can be done by customizing its source code to a configurable incremental update, but it is better to consider the table structure change, whether to delete the full table rebuild, or how to handle it)

2, because the table is instantaneous, it can not be provided to other parties at the same time to use.

Refer to their website description and git

https://issues.apache.org/jira/browse/KYLIN-3044

Set the default degree of parallelism for sqoop imports

kylin2.3 version enables JDBC data sources (you can generate hive tables directly from SQL, eliminating the hassle of manually conducting data to hive and building hive tables)