Kafka cluster expansion and redistribution of partitions

Go from: Kafka cluster expansion and redistribution of partitions

We add the machine to the already deployed Kafka cluster is the most normal demand, and add it is very convenient, we need to do is to copy the corresponding configuration file from the deployed Kafka node, and then change the broker ID inside to be globally unique, Finally, this node is launched to add it to an existing Kafka cluster.

But the problem is that the newly added Kafka node does not automatically allocate data, so the load on the cluster cannot be shared unless we create a new topic. But now we want to manually move some of the partitions to the newly added Kafka node, and Kafka internally provides the tools to redistribute a topic partition. Before we redistribute the topic partition, let's take a look at the distribution of the various partitions that are now topic:

.  /bin/kafka-topics. SH--topic iteblog--describe--zookeeper www.iteblog.com:2181 topic:iteblog partitioncount:7 Replicationfactor:2 Configs:Topic:iteblog partition:0 leader:1 replicas:1,2 isr:1,2 topic:iteblog partitio   N:1 leader:2 replicas:2,3 isr:2,3 topic:iteblog partition:2 leader:3 replicas:3,4 isr:3,4 topic:iteblog Partition:3 leader:4 replicas:4,1 isr:4,1 topic:iteblog partition:4 leader:1 replicas:1,3 isr:1,3 Topi C:iteblog partition:5 leader:2 replicas:2,4 isr:2,4 topic:iteblog partition:6 leader:3 replicas:3,1 isr:3 , 1

As you can see from the above output, there are 7 partitions in the Iteblog theme, but we have only 4 brokers, so some brokers will be able to maintain more partitions. Now we add a Kafka node on top of the existing cluster, and then redistribute the partition using the Kafka-reassign-partitions.sh tool that comes with Kafka. The tool has three modes of use:

1, generate mode, given the need to redistribute the topic, automatically generate reassign plan (not executed)
2. Execute mode, reassign partition according to the specified reassign plan
3, verify mode, verify the reallocation of partition is successful

Now we need to redistribute the partitions that were originally distributed on the broker 1-4 node to the broker 1-5 node and generate the reassign plan with the Kafka-reassign-partitions.sh tool, but first we have to define a file as required. It shows which topic need to be re-partitioned and the contents are as follows:

[Iteblog@www.iteblog.com ~]$ cat Topics-to-move.json {"Topics": [{"topic": "Iteblog"}],