High-throughput distributed publishing subscription messaging system kafka--management Tools Kafka Manager

I. Overview

Kafka is used by many teams within Yahoo, and the media team uses it to do a real-time analysis pipeline that can handle peak bandwidth of up to 20Gbps (compressed data).
To simplify the work of developers and service engineers in maintaining the Kafka cluster, a web-based tool called the Kafka Manager was built, called Kafka Manager. This management tool can easily find out which topic distributed in the cluster are uneven, or if the partitions are unevenly distributed across the cluster. It supports managing multiple clusters, selecting replicas, re-allocating replicas, and creating topic. At the same time, this management tool is also a very good tool to quickly browse the cluster.
The software is written in the Scala language. Currently (February 03, 2015) Yahoo has open source Kafka Manager tool. The Kafka Cluster Management tool supports the following features:
1, management of several different clusters;
2, it is easy to check the status of the cluster (topics, brokers, copy distribution, partition distribution);
3, select a copy;
4, generating partition allocation (Generate partition assignments) based on the current state of the cluster;
5. Reallocate partitions.

Second, Kafka manager download and installation

Project Address: Https://github.com/yahoo/kafka-manager

This project is more useful than https://github.com/claudemamo/kafka-web-console, the information displayed is richer, and the Kafka-manager itself can be a cluster.

However, Kafka-manager also does not have permission management capabilities.

Download:

git clone [email protected]:yahoo/kafka-manager.git

After downloading, only energy code you can not do anything, we want to compile the project package, the software is written in the Scala language, it is difficult to compile the package, he relies on SBT. SBT is more difficult to install.

Three, SBT installation

1. Download sbt-0.13.11

My server is CentOS auto install several times without success, I still choose to install manually.

Please go to http://www.scala-sbt.org/download.html to download the latest version, my version is sbt-0.13.11

2, set up the directory, unzip the file to the established directory

$ sudo mkdir/opt/scala/sbt$ sudo tar zxvf sbt-0.13.11.tgz-c/opt/scala/

3. Create a script file to start SBT

/* Select a location to create a script text file that starts SBT, such as a new text file named SBT under the/opt/scala/sbt/directory */$ cd/opt/scala/sbt/$ vim sbt/* add bt_opts= "to the SBT text file"- xms512m-xmx1536m-xss1m-xx:+cmsclassunloadingenabled-xx:maxpermsize=256m "Java $SBT _opts-jar/opt/scala/sbt/bin/ Sbt-launch.jar "[email protected]" and press ESC to enter: Wq save exit, note that the path in the red font can be an absolute path or a relative path. Just locate the Sbt-launch.jar file in the extracted SBT file package correctly */

and modify the SBT file permissions

4. Configure the PATH environment variable to ensure that the SBT command can be used in the console

$ vim/etc/profile/* After adding the following code at the end of the file, save exit */export path=/opt/scala/sbt/: $PATH

　　

/* Make the configuration file effective immediately */$ Source/etc/profile

5. Test if SBT is installed successfully
In the first execution, some packages will be downloaded before they can be used properly, and the process of downloading is very slow. After successful installation, the following is displayed

SBT Sbt-version
[INFO] Set current project-to-SBT (in Build file:/opt/scala/sbt/)
[INFO] 0.13.11

　　

Four, Yi Packaging

CD KAFKA-MANAGERSBT Clean dist

The resulting package will be under Kafka-manager/target/universal. The generated package only requires a Java environment to run, and no SBT is required on the deployed machine.

If packaging will be slow to be a little patience, there may be a package failure, you can consider configuring the agent.

Iv. Kafka Manager Deployment

1, a good package, in the deployment of the machine to decompress, modify the configuration file, you can run the--decompression

Unzip Kafka-manager-1.0-snapshot.zip

2, modify the conf/application.conf, change the kafka-manager.zkhosts to their own zookeeper server address

Kafka-manager.zkhosts= "192.168.1.237:2181"

3. Start

CD kafka-manager-1.0-snapshot/bin./kafka-manager-dconfig.file=. /conf/application.conf

4. View Help and background run

./kafka-manager-hnohup./kafka-manager-dconfig.file=. /conf/application.conf >/dev/null 2>&1 &  

Description: Normally, the play framework should automatically load the contents of the conf/application.conf configuration, but it seems that this does not work, explicitly specify the line.

Reference: HTTPS://GITHUB.COM/YAHOO/KAFKA-MANAGER/ISSUES/16

5, the default HTTP port is 9000, you can modify the value of the Http.port in the configuration file, or pass the command line parameters:

V. SBT Configuration Agent

SBT Configuration HTTP Proxy reference documentation: Http://www.scala-sbt.org/0.12.1/docs/Detailed-Topics/Setup-Notes.html#http-proxy

You can set the three numbers by using-D:

Java-dhttp.proxyhost=myproxy-dhttp.proxyport=8080-dhttp.proxyuser=username-dhttp.proxypassword=mypassword

You can also set the SBT_OPTS environment variable in the following way:

Export sbt_opts= "$SBT _opts-dhttp.proxyhost=myproxy-dhttp.proxyport=myport"

Note: myproxy, do not take the HTTP prefix or the port number in this value.

For example, your agent is http://localhost:8123, so you should configure this:

Export sbt_opts= "$SBT _opts-dhttp.proxyhost=localhost-dhttp.proxyport=8123"

High-throughput distributed publishing subscription messaging system kafka--management Tools Kafka Manager