Kafka Distributed construction

Kafka Distributed construction

(192.168.230.129)master(192.168.230.130)slave1(192.168.230.131)salve2在master、slave1、slave2三台主机上配置kafaka分布式集群

Preparation: Configure the Zookeeper1 on three machines, unzip the Kafka compressed file to the specified directory.

[[email protected] software]# tar -zxf kafka_2.10-0.8.1.1.tgz -C /opt/modules

2. Modify the Server.properties file in the/opt/modules/kafka_2.10-0.8.1.1/config directory

[[email protected] ~]# cd /opt/modules/kafka_2.10-0.8.1.1/config[[email protected] config]# vi server.properties

broker.id=1zookeeper.connect=master:2181,slave1:2181,slave2:2181

3, the master host on the Kafka configuration of the installation directory sent to SLAVE1, Salve2 host

[[email protected] config]# scp -r /opt/modules/kafka_2.10-0.8.1.1/ [email protected]:/opt/modules/kafka_2.10-0.8.1.1/[[email protected] config]# scp -r /opt/modules/kafka_2.10-0.8.1.1/ [email protected]:/opt/modules/kafka_2.10-0.8.1.1/

4. Modify the/opt/modules/kafka_2.10-0.8.1.1/config/server.properties configuration file in the SLAVE1 host

[[email protected] kafka_2.10-0.8.2.1]# vi config/server.propertiesbroker.id=2

5. Modify the/opt/modules/kafka_2.10-0.8.1.1/config/server.properties configuration file in the SLAVE1 host

[[email protected] kafka_2.10-0.8.1.1]# vi config/server.propertiesbroker.id=3

6, modify the three hosts in the/opt/modules/kafka_2.10-0.8.1.1/config/server.properties configuration file

[[email protected] kafka_2.10-0.8.1.1]# vi config/server.properties#host.name=localhost将#注释去掉，并改成master主机下：host.name=master

[[email protected] kafka_2.10-0.8.1.1]# vi config/server.propertiesslave1主机下：host.name=slave1

[[email protected] kafka_2.10-0.8.1.1]# vi config/server.propertiesslave2主机下：host.name=slave2

7. Start broker on each node

To start the broker on the master host:

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-server-start.sh config/server.properties

To start the broker on the SLAVE1 host:

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-server-start.sh config/server.properties

To start the broker on the SLAVE2 host:

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-server-start.sh config/server.properties

8. Create a topic in the Kafka cluster

Create a topic in master Host

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-topics.sh --create --zookeeper master:2181,slave1:2181,slave2:2181 --replication-factor 3 --partitions 3 --topic chinesescore

9. See if the message was created successfully

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-topics.sh --list --zookeeper master:2181,slave1:2181,slave2:2181 --topic chinesescore

10. View a topic partition and copy status information

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-topics.sh --describe --zookeeper master:2181,slave1:2181,slave2:2181 --topic chinesescore

11. Write a message to a topic with a producer

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-console-producer.sh --broker-list  master:9092,slave1:9092,slave2:9092 --topic chinesescore

12. Use a comsumer to read information from a topic

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-console-consumer.sh --zookeeper master:2181,slave1:2181,slave2:2181 --from-beginning --topic chinesescore

13. View topic from Zookeeper

[[email protected] kafka_2.10-0.8.1.1]# zkCli.sh

[zk: localhost:2181(CONNECTED) 4] ls /brokers/topics

13 writing Kafka producer programs using Java:

Use a comsumer to read information from a topic

[[email protected] kafka_2.10-0.8.1.1]$ ./bin/kafka-console-consumer.sh --zookeeper hadoop-sparse:2181 --from-beginning --topic orders

14. Close Kafka Service

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-server-stop.sh

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-server-stop.sh

[[email protected] kafka_2.10-0.8.1.1]# ./bin/kafka-server-stop.sh

Kafka Distributed construction