Kafka cluster expansion and data migration

A Kafka cluster expansion is relatively simple, machine configuration is the same premise only need to change the configuration file in the Brokerid to a new start up. It is important to note that if the company intranet DNS changes are not very timely, the old machine needs to be added to the new server host, otherwise the controller server from ZK to get the domain name but not resolve the new machine address situation.

Two after the cluster expansion, the data is not automatically balanced to the new machine, need to use kafka-reassign-partitions.sh this tool script. The script can work in three modes--generate,--execute,--verify

Used to generate topic migration Plan files, carry out migration plan, review the progress of the migration, official documents written very clear, the lack of the place is that each step requires manual execution is more troublesome and error-prone, borrowed from the group before the students wrote

A script is almost semi-automated or very efficient, forget about it.

startrebalance.sh

#!/bin/bash

source/etc/profile

source./conf.sh

echo "zkconf:" $zkServer $clustername

echo "Brokerlist:" $brokerIdList

echo "########################## #start generate Reblance conf##############################"

Content= ' kafka-reassign-partitions.sh--zookeeper $zkServer $clustername--topics-to-move-json-file Topicmove.conf.multi--broker-list $brokerIdList--generate '

Content= ' echo $content | Awk-f ' proposed partition reassignment configuration ' {print $} '

Echo $content

echo $content > ressgintopic.conf

Echo $content >>./log/reblance.log

echo "" >>./log/reblance.log

echo "########################## #end Generate Reblance conf##############################"

#start reblance

echo "########################## #start reblance##############################"

kafka-reassign-partitions.sh--zookeeper $zkServer $clustername--reassignment-json-file ressgintopic.conf--execute

conf.sh

#!/bin/bash

zkserver=1.1.1.1:2181

Clustername=/kafka/chroot

brokeridlist=1,2,3

Kafka cluster expansion and data migration