Kafka Cluster deployment

Kafka is a distributed message queue and needs to rely on zookeeper, please install the ZK cluster first.

Kafka installation Package Decompression

tar XF kafka_2. Ten-0.9. 0.1  mv kafka_2. Ten-0.9. 0.1 /usr//usr/kafka

Configuration file

server.properties# theIDof the broker. This must is set to a unique integer forEach broker.broker.ID=0# Zookeeper Connectionstring(See Zookeeper Docs fordetails). # A Comma separated host:port pairs, each corresponding to a zk# server. e.g."127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # can also append an optionalchroot stringTo the URLs to specify the# root directory forAll Kafka Znodes.zookeeper.connect=node1:2181, Node2:2181, Node3:2181

The plan has 3 nodes and the broker's ID should be different
Node1 to Broker.id=0
Node2 to Broker.id=1
Node3 to broker.id=2

Copy Kafka directory to other nodes on Node1

SCP -r/usr/kafka node2:/usr/SCP -r/usr/kafka node3:/usr/

Replace Broker.id on Node2, Node3

sed ' s/broker.id=.*/broker.id=1/ ' /usr/kafka/config/sed's/broker.id=.*/broker.id=2/' /usr/ Kafka/config/server.properties

Environment variables can be configured, optional

Export kafka_home=/usr/Kafkapath= $PATH: $KAFKA _home/bin

Start Kafka
1. Start Zookeeper on 3 nodes
2. Start Kafka on 3 nodes

$ bin/kafka-server-start. SH config/server.properties

Test
Create a topic and use kafka-topics.sh

$ kafka-topics. sh --zookeeper node1:2181, Node2:2181, node3:218123 --  Kafka-topics. sh --zookeeper node1:2181, Node2:2181, node3: 2181--kafka-topics. sh --zookeeper node1:2181, Node2:2181, node3:2181 --describe--topic test

Creating generators and consumers
To turn on the creator on any node

$ bin/kafka-console-producer. sh --broker-list node1:9092, Node2:9092, node3:9092 --topic test

Multiple consumers can be opened on multiple nodes

$ bin/kafka-console-consumer. sh --zookeeper node1:2181, Node2:2181, node3:2181

--from-beginning represents the data that gets the queue from the earliest start
After consuming a few data, execute the following statement to see if it starts from scratch, and the different partition return data unordered

$ bin/kafka-console-consumer. sh --zookeeper node1:2181, Node2:2181, node3:2181 --topic test

The returned data is unordered.

Kafka Cluster deployment