Kafka cluster installation and resizing

Introduction

Cluster installation:

I. preparations:

1. Version introduction:

Currently we are using a version of kafka_2.9.2-0.8.1 (scala-2.9.2 is officially recommended for Kafka, in addition to 2.8.2 and 2.10.2 available)

2. Environment preparation:

Install JDK 6. The current version is 1.6 and java_home is configured.

3. Configuration modification:

1) copy the online configuration to the local Kafka directory.

2) The broker and IP address in server. properties must be unique.

3) log. dirs must be manually specified in server. properties. This configuration item is the storage directory of logs received by Kafka (currently we save 7 days of data log. Retention. Hours = 168)

4) zookeeper can be specified in the following format. You can manually customize the zookeeper directory and use commas to separate IP addresses.

Zookeeper. Connect = 1.1.1.1: 2181, 1.1.1.2: 2181, 1.1.1.3: 2181/kafka-0.8.1

5) The default number of partitions and backups is 10 and 3 (Num. partitions = 10/Default. Replication. Factor = 3). Of course, you can also manually specify this when creating a topic.

For detailed configuration items, see the link: Kafka Configuration

4. Deployment:

Currently, Kafka is installed in/opt/cmd_install.

Jmx_port = 9997 bin/kafka-server-start.sh config/server. Properties>/dev/null 2> & 1 &

Deploy a new node and run the command to start Kafka. jmx_port is used to open the manageable port of Kafka (disabled by default ).

5. command:

Common addition, deletion, modification, and query commands: Basic Kafka commands and advanced commands

6. Resizing:

1) first deploy the new node environment, modify the configuration according to the "modify configuration" above, and then start the cluster to ensure that the new node is available.

2) The expanding your cluster section of the "advanced command" describes the basic resizing methods:

1. Generate the JSON file used for resizing:
   Cat topics-to-move.json
   

   {"Topics": [{"topic ":"Topic1"}], // The bold part is the name of the topic to be manually specified
   "Version": 1
   }

2. Use the JSON file written in the previous step and the Kafka command to generate the data migration configuration.
   Bin/kafka-reassign-partitions.sh -- topics-to-Move-JSON-file topics-to-move.json -- zookeeper 1.1.1.99: 2181/kafka-0.8.1 -- broker-list"0, 1, 2, 3, 4"-- Generate // specifies the brokers to which data is migrated.
   

3. Save the first part as a back-up (that is, the JSON string above proposed partition reassignment configuration), the following JSON string is the configuration to be used for resizing, save it as a expand-cluster-reassignment.json

4. Run the scale-up command: Bin/kafka-reassign-partitions.sh -- zookeeper 1.1.1.99: 2181/kafka-0.8.1 -- reassignment-JSON-file expand-cluster-reassignment.json -- If execute is normal, a JSON string of the same type is generated, indicating the original and target States

5. Query execution status: Bin/kafka-reassign-partitions.sh -- zookeeper 1.1.1.99: 2181/kafka-0.8.1 -- reassignment-JSON-file expand-cluster-reassignment.json -- after verify runs properly, the current data migration does not require partion will be returned, information status similar to below
   

   Reassignment of partition [foo1, 0] completed successfully // moved successfully reassignment of partition [foo1, 1] is in progress // This line indicates that data is being moved. Reassignment of partition [foo1, 2] is in progressreassignment of partition [foo2, 0] completed partition of partition [foo2, 1] completed successfully reassignment of partition [foo2, 2] completed successfully

6. Once data migration is started, it cannot be stopped or forcibly stopped. This will cause data inconsistency and irreparable consequences.

7. Note: The principle of Kafka data migration is to first copy the data to the target node and then delete the data of the original node. In this case, if the original Node space of the cluster is insufficient, do not specify it as the Migration broker. This will exhaust the space of the original node. For example, if the original node is broker 0, 1, 2, 3, 4. Do not specify -- broker-list"0, 1, 2, 3, 4", It should be -- broker-list"5, 6".
   In addition, data migration can also be customized manually.