On Kylin1.6 streaming Kafka cube build in the process of success encountered in the pit

Look at the Kylin official website (http://kylin.apache.org/docs16/tutorial/cube_streaming.html) in the understatement, it seems as long as the step by step to do it, you can quickly run through and get the fruit of expectation. But, and then the egg ... This practice let me very soberly understand, in the open source world, really is the version too many, bug too many, pit too many ...

If you just read the document 100 times, but not practice it, then perhaps reading comprehension level will improve a lot. In order to go through this operation, and get the expected results, incredibly cost ten days! But also may be as just contact open source Platform 1 months of my problem analysis ability lacks!

Pit 1: Official documentation does not focus on displaying instructions that are more recent! The commands in the document do not apply to older versions of Kylin (cost 2 days)

I started this practice from the beginning of December, when the company installed or Kylin1.5. But it happens that these days, the latest kylin1.6 released! Natural official documents (http://kylin.apache.org/docs16/tutorial/cube_streaming.html) have also been updated to the latest version. However, the beginning of the document does not clearly alert you to this point! Because Kylin1.6 has made great changes to streaming support based on Kylin1.5, such as the change of build streaming cube command (the SH command in kylin1.5 is deprecated). So obviously, when I use the Kylin1.6 command to execute on the installed Kylin1.5, I always report some Nosuchmethod errors! Well, fortunately for this mistake a bit of feeling, vaguely feel that this is because the version is incompatible with the reason. From discovering problems to upgrading to Kylin1.6, it took about 2 days!

Different versions of Kylin build streaming the difference between commands of the cube:
Kylin1.6:
curl-x PUT--user admin:kylin-h "Content-type:application/json;charset=utf-8"-d ' {"Sourceoffsetstart": 0, "sourceoffsetend": 9223372036854775807, "BuildType": "BUILD"} ' http://192.168.18.12:7070/ Kylin/api/cubes/streamingcube12/build2
Description: I just use this command against the installation of the kylin1.5 to run, always error, so I confused the N-long!
Kylin1.5:
$ kylin_home/bin/streaming_build.sh streaming_cube 300000 0

Pit 2: Official documents can only be used for reference, not as a canon! Too serious, you lose! (Cost 1 days)
As below, I copy the command from the official website to use the name of the Startsourceoffset and Endsourceoffset, the operation of the process is directly error. Carefully compare the name in the command with the name of the error prompt in the CLI sourceoffsetstart and sourceoffsetend can find subtle differences. Thus, the official website of the document is only for reference, not superstitious! If you take it too seriously, you lose! (about the Official document content this piece of
Issue, I also found a bug in the process of cleaning up and maintaining KYLIN hbase files, see https://issues.apache.org/jira/browse/KYLIN-2261)

Error from CLI:

Pit 3:kylin1.6 + Kafka1.0 can work, Kylin1.6 + Kafka0.9 has a bug (cost 2 days)

Perhaps it is because the previous question is a bit impatient, perhaps because it habitually ignores the non-practice part of the document. So go straight from the operation step and start working again. But after the final submission of the Curl command, and directly reported the following javalangnullpointer or nosuchmethoderror errors, but also TMD in these two errors randomly appeared! The view is very big!!! Last resort, submitted a issue to KYLIN official commnuity (https://issues.apache.org/jira/browse/KYLIN-2262?filter=-2). The official reproduce this error and basically confirms the fact that the version is incompatible. This process, it took me almost two days! This communication with the official is always so inefficient, but fortunately the direction of the search for answers is not mistaken!

Pit 4: Cluster resource issues (cost 2 days)
The company's work with the CDH can not be rushed to upgrade Kafka components, under the leadership of the proposal, I have to build their own environment. Since there is no other CDH test environment available at the moment, make the first use of a personal computer (configuration can be 12g+ four core +i7) download a CDH OVF virtual machine to do the next Test verification. Everything is ready, to the last step when you submit the build cube, and then directly error
java.lang.InterruptedException (https://issues.apache.org/jira/browse/KYLIN-2278)!

No words! The next day after submitting the issue, nobody. Can only guess is because the local configuration is too low, have to use the company's stick a little machine, build CDH environment and install Kylin1.6 + Kafka1.0.

Pit 5: Because of the inexplicable Hadoop (originally a cluster of three machines, but only one machine can be used, so the engineer changed to a single-node mode), the build cube's Mr Job has been in the pending state.

The heart is cool, why every step is not smooth? Still submitted a issue (https://issues.apache.org/jira/browse/KYLIN-2281), but still empty and so on. Of course, in the process of waiting to consult with the engineer, decided to re-install Linux to the server, eliminate all possible system environment problems, reinstall cdh5.7+kylin1.6+kafka1.0. (Cost 2 days)

such as 1, is the job status seen on the monitor page of Kylin. Nnd,duration so long, but progress is still 0%.

Like 2, on the Hadoop monitor page, you can see it in the pending process, although I repeatedly delete-Create multiple

6. After a while, we finally got a new environment and referenced the document (Https://kyligence.gitbooks.io/kap-manual/content/zh-cn/quickstart/quickstart_ cdh.cn.html), the yarn has been partially set as follows,

Then submit the order, finally, finally see the expected results!!! (Cost 1 day)

Command:

Curl-x PUT--user admin:kylin-h "Content-type:application/json;charset=utf-8"-d ' {"Sourceoffsetstart": 0, "sourceOffs Etend ": 9223372036854775807," BuildType ":" BUILD "} ' http://192.168.18.62:7070/kylin/api/cubes/StreamingCube32/build2

Thanks GOD!!!

On Kylin1.6 streaming Kafka cube build in the process of success encountered in the pit