windows下kafka源碼閱讀環境搭建

標籤：kafka 源碼 環境 搭建 windows

1. 工具準備：jdk1.8,scala-2.11.11,gradle-3.1,zookeeper-3.4.5,kafka-0.10.0.1-src.tgz, kafka_2.11-0.10.0.1.tgz

2. 安裝jdk

3. 安裝scala

4. 搭建zookeeper

5. kafka源碼構建

   解壓kafka-0.10.0.1-src.tgz，命令列進行kafka-0.10.0.1-src，執行gradle idea

   
   

   650) this.width=650;" src="https://s5.51cto.com/oss/201711/05/e7b90ed54a50810c84f71b7b03bc4dbc.png-wh_500x0-wm_3-wmp_4-s_3555752997.png" title="1.png" alt="e7b90ed54a50810c84f71b7b03bc4dbc.png-wh_" />

註：

1)gradle idea，用國外的庫構建的下載速度很慢，在進入項目裡面，找到build.gradle檔案的修改庫為阿里鏡像庫

buildscript {

  repositories {

        maven{ url ‘http://maven.aliyun.com/nexus/content/groups/public/‘}

    }

  apply from: file(‘gradle/buildscript.gradle‘), to: buildscript

  dependencies {

    // For Apache Rat plugin to ignore non-Git files

    classpath "org.ajoberstar:grgit:1.5.0"

    classpath ‘com.github.ben-manes:gradle-versions-plugin:0.12.0‘

    classpath ‘org.scoverage:gradle-scoverage:2.0.1‘

  }

}

allprojects {

  apply plugin: ‘idea‘

  apply plugin: ‘eclipse‘

  apply plugin: "jacoco"

  allprojects {

    repositories {

        maven{ url ‘http://maven.aliyun.com/nexus/content/groups/public/‘}

    }

}

2)執行gradle idea時，報錯

* What went wrong:  

A problem occurred evaluating root project ‘kafka-0.10.0.1-src‘.  

> Failed to apply plugin [class ‘org.gradle.api.plugins.scala.ScalaBasePlugin‘]  

   > No such property: useAnt for class: org.gradle.api.tasks.scala.ScalaCompileOptions  

  

* Try:  

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.  

  

BUILD FAILED  

解決方案：開啟kafka-0.10.0.1-src目錄下build.gradle檔案

添加如下行

[plain] view plain copy

ScalaCompileOptions.metaClass.daemonServer = true  

ScalaCompileOptions.metaClass.fork = true  

ScalaCompileOptions.metaClass.useAnt = false  

ScalaCompileOptions.metaClass.useCompileDaemon = false 

7.用intellij idea開啟kafka-0.10.0.1-src

650) this.width=650;" src="https://s4.51cto.com/oss/201711/05/2e54b08ee3d7b2412c280f36f31cd0bd.png-wh_500x0-wm_3-wmp_4-s_2374849679.png" title="2.png" alt="2e54b08ee3d7b2412c280f36f31cd0bd.png-wh_" />

8.複製log4j.properties檔案放到core模組的src/main/scala路徑下，這樣才能正常的顯示日誌

650) this.width=650;" src="https://s4.51cto.com/oss/201711/05/2ed43d098b7cebdf2faa21876ad0dac4.png-wh_500x0-wm_3-wmp_4-s_3605728480.png" title="3.png" alt="2ed43d098b7cebdf2faa21876ad0dac4.png-wh_" />

9.增加scala-sdk

650) this.width=650;" src="https://s1.51cto.com/oss/201711/05/0c3b177e184622b505534f8879c596fc.png-wh_500x0-wm_3-wmp_4-s_88991643.png" title="4.png" alt="0c3b177e184622b505534f8879c596fc.png-wh_" />

10.配置啟動參數

650) this.width=650;" src="https://s4.51cto.com/oss/201711/05/12e18887649e11e1f9db69d044850dc6.png-wh_500x0-wm_3-wmp_4-s_1139839183.png" title="5.png" alt="12e18887649e11e1f9db69d044850dc6.png-wh_" />

11.啟動項目

650) this.width=650;" src="https://s3.51cto.com/oss/201711/05/ce3716f9bb285fd7a644b66b2441c228.png-wh_500x0-wm_3-wmp_4-s_475031007.png" title="6.png" alt="ce3716f9bb285fd7a644b66b2441c228.png-wh_" />

12.驗證

解壓kafka_2.11-0.10.0.1.tgz檔案，命令列進入bin/windows

1）建立topic

C:\webserver\kafka_2.11-0.10.0.1\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --

partitions 1 --topic testMsg

Created topic "testMsg".

控制台日誌如下：

650) this.width=650;" src="https://s5.51cto.com/oss/201711/05/38fb278207d888d2f482d0fced205b5d.png-wh_500x0-wm_3-wmp_4-s_3427900052.png" title="7.png" alt="38fb278207d888d2f482d0fced205b5d.png-wh_" />

2）執行生產者命令，產生訊息：

650) this.width=650;" src="https://s5.51cto.com/oss/201711/05/d95d2efbc0fb9428444026348165dd22.png-wh_500x0-wm_3-wmp_4-s_3822918000.png" title="8.png" alt="d95d2efbc0fb9428444026348165dd22.png-wh_" />

3）執行消費者命令，消費訊息：

650) this.width=650;" src="https://s1.51cto.com/oss/201711/05/c81d90a057bfd0c7c3221db7726f6f21.png-wh_500x0-wm_3-wmp_4-s_2851271549.png" title="9.png" alt="c81d90a057bfd0c7c3221db7726f6f21.png-wh_" />

問題：

ERROR Processor got uncaught exception. (kafka.network.Processor)
java.nio.BufferUnderflowException

解決方案：

這是用戶端的jar包和叢集的版本號碼在小版本不一致導致，保持版本號碼一致即可。

scala-2.11.11,kafka-0.10.0.1-src.tgz, kafka_2.11-0.10.0.1.tgz

本文出自 “柯浩堅” 部落格，請務必保留此出處http://gauyanm.blog.51cto.com/629619/1979198

windows下kafka源碼閱讀環境搭建