Kafka cluster Installation (CentOS 7 environment)

Introduction of environment operating system and software version

1. Environment operating system for CentOS Linux release 7.2.1511 (Core)

Available Cat/etc/redhat-release queries

2. Software version

The Kafka version is: 0.10.0.0

Second, the basic preparation of software

Because the Kafka cluster needs to rely on the zookeeper cluster for co-management, the ZK cluster needs to be built beforehand. This article mainly introduces the building of Kafka cluster environment.

Three, detailed installation and construction steps

1. Download the compressed package kafka_2.10-0.10.0.0.tgz to the/data/soft directory

2. Extract the kafka_2.10-0.10.0.0.tgz to the/data/app/kafka directory

Tar–xzf Kafka_2.10-0.10.0.0.tgz–c/data/app/kafkacluster

Rename the folder to 19092, go to config directory, modify the Server.properties file

3. Open server.properties with the VI command

1 [[email protected] config]# VI server.properties

4. Modify the following:

 
1 broker.id=0
2 port=19092
3 log.dirs=/data/app/kafkacluster/19093/bin/kafka-logs19092
4 zookeeper.connect=192.168.1.18:3001,192.168.1.18:3002,192.168.1.18:3003

Kafka on the other two servers, first modify the folder name (the other two folder names are 19093 and 19094 in this article)

Then go to config directory and change server.properties name to Server1.properties and Server2.properties respectively.

The configuration in server1.properties needs to be changed:

 
 
1 broker.id=1
2 port=19093
3 log.dirs=/data/app/kafkacluster/19093/bin/kafka-logs19093
4 zookeeper.connect=192.168.1.18:3001,192.168.1.18:3002,192.168.1.18:3003

The configuration in server2.properties needs to be changed:

 
1 broker.id=2
2 port=19094
3 log.dirs=/data/app/kafkacluster/19094/bin/kafka-logs19094
4 zookeeper.connect=192.168.1.18:3001,192.168.1.18:3002,192.168.1.18:3003

Iv. Start kafka& test verification

1. Start the independent ZK cluster first and start all three (./zkserver.sh start)

2. Go to the Bin directory of Kafka and start the service./kafka-server-start.sh. /config/server.properties (three servers to start)

 
1 ./kafka-server-start.sh ../config/server1.properties
2 ./kafka-server-start.sh ../config/server2.properties

In addition, when the other node is started, the node that starts first will display the information record added by the other nodes as follows:

 
1 [2017-01-18 14:44:24,352] INFO Partition [aaa,0] on broker 0: Expanding ISR for partition [aaa,0] from 0 to 0,1 (kafka.cluster.Partition)
2 [2017-01-18 14:44:37,065] INFO Partition [aaa,0] on broker 0: Expanding ISR for partition [aaa,0] from 0,1 to 0,1,2 (kafka.cluster.Partition)

3. Verify the startup process

 
1 [[email protected] bin]# jps
2 25778 Kafka
3 26132 Jps
4 25285 Kafka
5 25014 QuorumPeerMain
6 25064 QuorumPeerMain
7 25531 Kafka
8 25116 QuorumPeerMain

4. Use the client to enter ZK

 
1 [[email protected] bin]# ./zkCli.sh -server 192.168.1.18:3001
2 Connecting to 192.168.1.18:3001

5. View directory Status

 
1 [zk: 192.168.1.18:3001(CONNECTED) 0] ls /
2 [controller_epoch, controller, brokers, zookeeper, test, admin, isr_change_notification, consumers, config]
3 [zk: 192.168.1.18:3001(CONNECTED) 1]

In the results shown above: Only zookeeper is zookeeper native, others are Kafka created

6. Create a topic:

 
1 [[email protected] bin]# ./kafka-topics.sh --create --zookeeper 192.168.1.18:3001,192.168.1.18:3002,192.168.1.18:3003 --replication-factor 3 --partitions 1 --topic test666
2 Created topic "test666".

7. Check the topic Status:

 
1 [[email protected] bin]# ./kafka-topics.sh --describe --zookeeper 192.168.1.18:3001,192.168.1.18:3002,192.168.1.18:3003 --topic test666
2 Topic:test666    PartitionCount:1    ReplicationFactor:3    Configs:
3     Topic: test666    Partition: 0    Leader: 0    Replicas: 0,2,1    Isr: 0,2,1

Output parameter explanation:

The first line is a description of all the partitions, and then each partition corresponds to one row, because we only have a single partition, so we add a row below.

Leader: Responsible for processing the read and write of messages, Leader is randomly selected from all nodes.

Replicas: Lists all replica nodes, regardless of whether the node is in the service

ISR: is a node in service

Visible from above, at this time the leader is 0

The following will kill 0 to see if leader changes

8. Send the message to test666:

 
1 [[email protected] bin]# ./kafka-console-producer.sh --broker-list localhost:19092,localhost:19093,localhost:19094 --topic test666
2 hello kafka!
3 hello littleMonster!
4 hello world!

9. Receive the message:

 
1 [[email protected] bin]# ./kafka-console-consumer.sh --zookeeper 192.168.1.18:3001,192.168.1.18:3002,192.168.1.18:3003 --topic test666 --from-beginning
2 hello kafka!
3 hello littleMonster!
4 hello world!

Message received successfully.

10. Find the process for the leader 0 and kill

1 [[email protected]/]# Ps-ef | grep ka

1 [[email protected]/]# Kill-925285

11. Check the topic status again:

 
1 [[email protected] bin]# ./kafka-topics.sh --describe --zookeeper 192.168.1.18:3001,192.168.1.18:3002,192.168.1.18:3003 --topic test666
2 Topic:test666    PartitionCount:1    ReplicationFactor:3    Configs:
3     Topic: test666    Partition: 0    Leader: 2    Replicas: 0,2,1    Isr: 2,1

Thus, in the ISR (node in service), 0 has disappeared and the newly elected leader is 2.

12. Send the message again

Receiving messages

The message is received normally, and the test passes.

Kafka cluster Installation (CentOS 7 environment)