Management Tools Kafka Manager

I. Overview

Kafka is used by many teams within Yahoo, and the media team uses it to do a real-time analysis pipeline that can handle peak bandwidth of up to 20Gbps (compressed data).
To simplify the work of developers and service engineers in maintaining the Kafka cluster, a web-based tool called the Kafka Manager was built, called Kafka Manager. This management tool can easily find out which topic distributed in the cluster are uneven, or if the partitions are unevenly distributed across the cluster. It supports managing multiple clusters, selecting replicas, re-allocating replicas, and creating topic. At the same time, this management tool is also a very good tool to quickly browse the cluster.
The software is written in the Scala language. Currently (February 03, 2015) Yahoo has open source Kafka Manager tool. The Kafka Cluster Management tool supports the following features:
1, management of several different clusters;
2, it is easy to check the status of the cluster (topics, brokers, copy distribution, partition distribution);
3, select a copy;
4, generating partition allocation (Generate partition assignments) based on the current state of the cluster;
5. Reallocate partitions.

Second, Kafka manager download and installation

Project Address: Https://github.com/yahoo/kafka-manager

This project is more useful than https://github.com/claudemamo/kafka-web-console, the information displayed is richer, and the Kafka-manager itself can be a cluster.

However, Kafka-manager also does not have permission management capabilities.

Download:

1	git clone git@github.com:yahoo/kafka-manager.git

After downloading, only energy code you can not do anything, we want to compile the project package, the software is written in the Scala language, it is difficult to compile the package, he relies on SBT. SBT is more difficult to install.

Three, SBT installation

1. Download sbt-0.13.11

My server is CentOS auto install several times without success, I still choose to install manually.

Please go to http://www.scala-sbt.org/download.html to download the latest version, my version is sbt-0.13.11

2, set up the directory, unzip the file to the established directory

12	$ sudo mkdir /opt/scala/sbt$ sudo tar zxvf sbt-0.13.11.tgz -C /opt/scala/

3. Create a script file to start SBT

1234567

/*选定一个位置，建立启动sbt的脚本文本文件，如/opt/scala/sbt/ 目录下面新建文件名为sbt的文本文件*/$ cd /opt/scala/sbt/$ vim sbt/*在sbt文本文件中添加BT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"java $SBT_OPTS -jar /opt/scala/sbt/bin/sbt-launch.jar "[email protected]"然后按esc键 输入 :wq 保存退出，注意红色字体中的路径可以是绝对路径也可以是相对路径，只要能够正确的定位到解压的sbt文件包中的sbt-launch.jar文件即可*/

and modify the SBT file permissions

1	$ chmod u+x sbt

4. Configure the PATH environment variable to ensure that the SBT command can be used in the console

123	$ vim /etc/profile/*在文件尾部添加如下代码后，保存退出*/export PATH=/opt/scala/sbt/:$PATH

　　

12	/*使配置文件立刻生效*/$ source /etc/profile

5. Test if SBT is installed successfully
In the first execution, some packages will be downloaded before they can be used properly, and the process of downloading is very slow. After successful installation, the following is displayed

1	sbt sbt-version<br>[info] Set current project to sbt (in build file:/opt/scala/sbt/)<br>[info] 0.13.11

　　

Four, Yi Packaging

12	cd kafka-managersbt clean dist

The resulting package will be under Kafka-manager/target/universal. The generated package only requires a Java environment to run, and no SBT is required on the deployed machine.

If packaging will be slow to be a little patience, there may be a package failure, you can consider configuring the agent.

Iv. Kafka Manager Deployment

1, a good package, in the deployment of the machine to decompress, modify the configuration file, you can run the--decompression

1	unzip kafka-manager-1.0-SNAPSHOT.zip

2, modify the conf/application.conf, change the kafka-manager.zkhosts to their own zookeeper server address

1	kafka-manager.zkhosts="192.168.1.237:2181"

3. Start

12	cd kafka-manager-1.0-SNAPSHOT/bin./kafka-manager -Dconfig.file=../conf/application.conf

4. View Help and background run

12	./kafka-manager -hnohup ./kafka-manager -Dconfig.file=../conf/application.conf >/dev/null2>&1&

Description: Normally, the play framework should automatically load the contents of the conf/application.conf configuration, but it seems that this does not work, explicitly specify the line.

Reference: HTTPS://GITHUB.COM/YAHOO/KAFKA-MANAGER/ISSUES/16

5, the default HTTP port is 9000, you can modify the value of the Http.port in the configuration file, or pass the command line parameters:

1	./kafka-manager -Dhttp.port=9001

V. SBT Configuration Agent

SBT Configuration HTTP Proxy reference documentation: Http://www.scala-sbt.org/0.12.1/docs/Detailed-Topics/Setup-Notes.html#http-proxy

You can set the three numbers by using-D:

1	java -Dhttp.proxyHost=myproxy -Dhttp.proxyPort=8080-Dhttp.proxyUser=username -Dhttp.proxyPassword=mypassword

You can also set the SBT_OPTS environment variable in the following way:

1	export SBT_OPTS="$SBT_OPTS -Dhttp.proxyHost=myproxy -Dhttp.proxyPort=myport"

Note: myproxy, do not take the HTTP prefix or the port number in this value.

For example, your agent is http://localhost:8123, so you should configure this:

1	export SBT_OPTS="$SBT_OPTS -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8123"

Management Tools Kafka Manager