Usage of Apache Kafka migration and resizing tools

Kafka migration and resizing tools

Site: https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool

Note:

When resizing a Kafka cluster, we need to meet the following requirements:

1. Migrate the specified topic to the new node in the cluster.
2. Migrate the specified partition of the topic to the new node.

1. migrate the topic to the new node. Assume that a Kafka cluster now runs three brokers, brokers. the ID is 101,102,103 in sequence. Later, due to the sudden increase of business data, three brokers need to be added, Broker. the IDS are 104,105,106 in sequence. the purpose is to migrate the push-Token-topic to the new node. The script (in JSON format) is as follows: [email protected]: $./bin/kafka-reassign-partitions.sh -- zookeeper 192.168.2.225: 2183/config/mobile/MQ/mafka
-- Topics-to-Move-JSON-file migration-push-token-topic.json -- broker-list "104,105,106" -- execute

The content of the script migration-push-token-topic.json file is as follows:

{
"Topics ":
[
{
"Topic": "Push-Token-topic"
}
],
"Version": 1
}

2. Number of replicats-factor copies

Lizh [email protected]: $./bin/kafka-reassign-partitions.sh -- zookeeper 192.168.2.225: 2183/config/mobile/MQ/mafka

-- Reassignment-JSON-file replicas-update-push-token-topic.json -- execute

If the push-Token-topic is a replica at the beginning, you need to change it to the 2 copy mode to improve availability.

The content of the script replicas-push-token-topic.json file is as follows:

{
"Partitions ":
[
{
"Topic": "log. mobile_nginx ",
"Partition": 0,
"Replicas": [101,102,104]
},
{
"Topic": "log. mobile_nginx ",
"Partition": 1,
"Replicas": [102,103,106]
},
{
"Topic": "XXXX ",
"Partition": Number,
"Replicas": [array]
}
],
"Version": 1
}

3. Topic Partition resizing usage

A. Expand the number of partitions first. The script is as follows:

For example, the initial number of push-Token-topic partitions is 12, which has been increased to 15

[Email protected]: $./bin/kafka-topics.sh -- zookeeper 192.168.2.225: 2183/config/mobile/MQ/mafka -- alter -- partitions 15 -- Topic push-Token-topic

B. Set the partition copy of the topic.

[Email protected]: $./bin/kafka-reassign-partitions.sh -- zookeeper 192.168.2.225: 2183/config/mobile/MQ/mafka

-- Reassignment-JSON-file partitions-Extension-Push-Token-topic. JSON -- execute

The content of the script partitions-extension-push-token-topic.json file is as follows:

{        "partitions":                [                {                        "topic": "push-token-topic",                        "partition": 12,                        "replicas": [101,102]                },                {                        "topic": "push-token-topic",                        "partition": 13,                        "replicas": [103,104]                },                {                        "topic": "push-token-topic",                        "partition": 14,                        "replicas": [105,106]                }                ],                     "version":1}