Idea under Kafka source reading compilation Environment construction

KafkaSource Compilation reading environment construction

Development Environment: Oracle Java 1.7.0_25 + idea + Scala 2.10.5 +gradle 2.1 + Kafka 0.9.0.1

First,GradleInstallation Configuration

Kafka code from 0.8.x Gradle gradle integrates and absorbs the maven > The main advantages are also overcome maven some limitations of itself -- You can access HTTPS ://www.gradle.org/downloads/  Download the latest Gradle version. Unzip the download to a directory, and then create an environment variable gradle_home point to the extracted directory, and then %gradle_home%\bin gradle installation configuration is ready. Open a cmd input gradle-v check it out:

650) this.width=650; "Src=" Http://s2.51cto.com/wyfs02/M00/8B/8F/wKioL1hRNM3RxXU_AAAb7OSI_ss634.png-wh_500x0-wm_3 -wmp_4-s_3005857787.png "style=" Float:none; "title=" 1.png "alt=" Wkiol1hrnm3rxxu_aaab7osi_ss634.png-wh_50 "/>

Second,KafkaSource code Download

install gradle after we start downloading kafka 0.10.0.1 Now we're using the 0.9.0.1 version, you can get from http://kafka.apache.org/downloads.html

650) this.width=650; "Src=" Http://s4.51cto.com/wyfs02/M00/8B/8F/wKioL1hRNQDCAnLVAADffla5kiw012.png-wh_500x0-wm_3 -wmp_4-s_1212188113.png "title=" 2.png "alt=" Wkiol1hrnqdcanlvaadffla5kiw012.png-wh_50 "/>

The role of the important directory is as follows:

BinCatalogue: Windowsand theUnixexecution scripts under the platform, such asKafka-server-start,Console-producer,Console-consumerwait
ClientsCatalogue: KafkaClient Code
ConfigCatalogue: Kafkaconfiguration file, where it is more important toServer.properties, startKafkabrokerneed to load this file directly
ContribCatalogue: Kafkawith theHadoopintegrated code, includingHadoop-consumerand theHadoop-producer
CoreCatalogue: Kafkathe core of the code, but also the key to learn after the author part
ExamplesCatalogue: Kafkasample code, such as how to use theJavaWrite a simpleProducerand theConsumer
System_test:system test scripts, mainly usedPythonWrite
other directories and configuration files are mostlyGradleconfiguration, do not repeat it.

Third, downloadGradle WrapperClass Library

separate this configuration as a step because the tutorial in the official website does not give a detailed configuration method. If you download the source code directly from the official website and execute the Gradlew Eclipse build project, you will get an error:

Error:could not find or load main Classorg.gradle.wrapper.GradleWrapperMain

in the Kafka Source code in the gradle subdirectory does not really have wrapper class library, so we have to install a gradle wrapper Library, the method is very simple, open a cmd window, in Gradle wrapper can be executed under the Kafka source code root directory . You only need to run this command once. It takes some time to download the required jar packages if it is a first time installation . Kafka 's source code is written in Scala and clearly shows that we are using the Scala version of 2.10.4. You can specify the Scala version in the gradle.properties file .

650) this.width=650; "Src=" Http://s4.51cto.com/wyfs02/M01/8B/93/wKiom1hRNRTwvCbhAAAY4ljPAKs730.png-wh_500x0-wm_3 -wmp_4-s_3815280791.png "title=" 3.png "alt=" Wkiom1hrnrtwvcbhaaay4ljpaks730.png-wh_50 "/>

after the command runs successfully, a wrapper subdirectory is included in the Gradle subdirectory of Kafka , which contains a jar package and a configuration file. this Gradle wrapper has been successfully installed on this machine:

650) this.width=650; "Src=" Http://s5.51cto.com/wyfs02/M00/8B/93/wKiom1hRNSKSIy4tAABGsRaarpo984.png-wh_500x0-wm_3 -wmp_4-s_3203591700.png "title=" 4.png "alt=" Wkiom1hrnsksiy4taabgsraarpo984.png-wh_50 "/>

Iv. GenerationIdeaproject files and import them intoIdeain

after all of this preparation is ready, we can now generate IdeaEngineering documents. The specific procedure is to open acmdwindow, switch toKafkaunder source code root path, runGradle Idea, if this is the first run, it may take some time to download the necessaryJarpackage, after waiting for some time, finally saw thebuildsuccessfulthe word indicates that the project project file was successfully generated:

650) this.width=650; "Src=" Http://s4.51cto.com/wyfs02/M00/8B/8F/wKioL1hRNTDwCzadAABII_C_hgo147.png-wh_500x0-wm_3 -wmp_4-s_3306286244.png "title=" 5.png "alt=" Wkiol1hrntdwczadaabii_c_hgo147.png-wh_50 "/>

Project into Idea Project

File-->open

650) this.width=650; "Src=" Http://s2.51cto.com/wyfs02/M01/8B/93/wKiom1hRNUKDlpSuAABtVOYhYKY716.png-wh_500x0-wm_3 -wmp_4-s_996908651.png "title=" 6.png "alt=" Wkiom1hrnukdlpsuaabtvoyhyky716.png-wh_50 "/>

View Source Engineering in Idea

650) this.width=650; "Src=" Http://s5.51cto.com/wyfs02/M01/8B/8F/wKioL1hRNVLzdzdhAABI59QSFyQ683.png-wh_500x0-wm_3 -wmp_4-s_1420646807.png "title=" 7.png "alt=" Wkiol1hrnvlzdzdhaabi59qsfyq683.png-wh_50 "/>

Five, configurationServer.properties

Kafka Broker start loading server.properties Kafka. Kafka.scala program Arguments. /config/server.properties kafka.scala The configuration file can be found.

also, because Kafka broker stores Some metadata information in zookeeper, you must first have a startup before you start Kafka broker. Zookeeper instance or cluster, and then we also need to update the zookeeper connection Information in the server.properties file ( hostname : Port CSV list )as shown in:

# Zookeeper Connection string (Seezookeeper docs for details).

# This was a comma separated host:portpairs, each corresponding to a ZK

# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".

# You can also append a optional chrootstring to the URLs to specify the

# root directory for all Kafka znodes.

zookeeper.connect=10.11.207.97:2181

# Timeout in MS for connecting to zookeeper

zookeeper.connection.timeout.ms=6000

Six, startKafka Broker

okay! now it 's time to say that everything is ready to run Kafka , and if all the previous steps are done correctly, a Kafka broker process should start up correctly, as shown in:

650) this.width=650; "Src=" Http://s4.51cto.com/wyfs02/M00/8B/93/wKiom1hRNWuR6HMTAAG8Yaveh58455.png-wh_500x0-wm_3 -wmp_4-s_748484423.png "title=" 8.png "alt=" Wkiom1hrnwur6hmtaag8yaveh58455.png-wh_50 "/>

In addition, if you do not see any log output at startup , you can copy the Log4j.properties file from the Config directory to the core\src\main\ Scala directory, you should be able to resolve this issue.

Well, you can use this operational environment to study Kafka the source code. Good luck to all of you :)

Seven,Log4j.propertiesFile path Settings

start Kafka Server It is strange that the log4j.properties file cannot be found and the following error is reported.

Log4j:warn No appenders could be found Forlogger (kafka.utils.VerifiableProperties).
Log4j:warn Initialize the log4j system properly.

You can only locate the file by placing the log4j.properties in the src/main/Scala Path, and then run the program to output the log information correctly.

after 7 steps, You can start the Kafka program correctly , carry out the relevant debug, and study its source code.

This article is from the "Digital Technology" blog, please be sure to keep this source http://7639538.blog.51cto.com/7629538/1882808

Idea under Kafka source reading compilation Environment construction