Kafka Data Migration

Scenario: The old cluster will no longer be used, the data in the Kafka cluster above is imported into the Kafka of the new cluster

Pour steps (for example, topic by day):

Because Kafka only retains 7 days of data by default, it only migrates data for nearly 7 days

1. First use the KAFKA-TOPIC.SH client to list all topic on the Kafka cluster

2. Remove the topic name for nearly 7 days, write to a txt text, each topic name line

3. Export: For I in $ (cat txt);d o kafka-console-consumer.sh--zookeeper=--topic= $i--from-beginning >> $i. Txt;done

4. Import: Create topic:kafka-topic.sh topic first

Cat $i. txt | kafka-conso-producer.sh--broker-list= ...--topic= $i

The second scenario is not commonly used:

Reference: http://www.cnblogs.com/dycg/p/3922352.html

The steps are as follows:

1. Build the Kafka service on the new node

Originally I had 2 machines, broker.id 1 and 2, respectively.

Now on my new machine The broker.id are set to 3 and 4 respectively

2. Start All Kafka Services

3. Confirm the topics to be moved

kafka-topics.sh--list--zookeeper 192.168.103.47:2181 View all topics

Copy these topic, and write the file in the following format, named Topics-to-move.json

{"Topics": [

{"topic": "Fortest1"},

{"topic": "Fortest2"},

{"topic": "Fortest3"}

],

"Version": 1

}

4. Create a mobile script

Run bin/kafka-reassign-partitions.sh--zookeeper 192.168.103.47:2181--topics-to-move-json-file Topics-to-move.json --broker-list "3,4"--generate

Where 3, 4 is the broker.id of your new node

This will generate a new set of JSON data

{"Version": 1, "Partitions": [{"topic": "Fortest1", "Partition": 0, "Replicas": [3,4]}, other parts omitted}

Write this string of JSON to the new file Reassignment-node.json

5. At this time, everything is ready to begin the migration

bin/kafka-reassign-partitions.sh--zookeeper 192.168.103.47:2181--reassignment-json-file Reassignment-node.json --execute

6. When appropriate, run the following command to view the results of the operation

bin/kafka-reassign-partitions.sh--zookeeper 192.168.103.47:2181--reassignment-json-file Reassignment-node.json-- Verify

Assumption appears

Error:assigned Replicas (3,4,0,1) don ' t match the list of replicas for reassignment (3,4) for partition [mpt-session,1]

Such a mistake, he is not really wrong, but is still in the replication of data.

After a while to run the Verify command, he will disappear (add to complete copy)

7. After the data has been migrated, the old service should not stop.

8. Modify all client producer or consumer connections to point to the new node.

9. After the test is normal,

This article from "After Tomorrow" blog, declined reprint!

Kafka Data Migration