Install and deploy a Kafka distributed cluster in Linux

Install and deploy a Kafka distributed cluster in Linux

Note: before deploying Kafka, deploy the JAVA and Zookeeper environments.

Prepare three CentOS_6.5_x64 servers:
IP: 192.168.0.249 dbTest249 Kafka IP: 192.168.0.250 Other250 Kafka IP: 192.168.0.20.webtest1_kafka

1. Install Kafka

# tar zxvf kafka_2.9.1-0.8.2.2.tgz -C /usr/local/# cd /usr/local/kafka_2.9.1-0.8.2.2/ 

2. We have installed three brokers on three machines: dbTest249, Other250, and webtest.pdf:
1. Edit dbTest249:

# vim config/server.properties                      

1 broker.id=0    2 port=9092    3 host.name=dbTest249    4 advertised.host.name=dbTest249    5 num.partitions=2    6 zookeeper.connect=dbTest249:2181,Other250:2181,webTest251:2181

2. Edit Other250

# vim config/server.properties                       

1 broker.id=1    2 port=9092    3 host.name=Other250    4 advertised.host.name=Other250    5 num.partitions=2    6 zookeeper.connect=dbTest249:2181,Other250:2181,webTest251:2181

3. Edit webtestbench

# vim config/server.properties

1 broker.id=2    2 port=9092    3 host.name=webTest251    4 advertised.host.name=webTest251    5 num.partitions=2    6 zookeeper.connect=dbTest249:2181,Other250:2181,webTest251:2181

3. Start the Kafka service on three machines: dbTest249, Other250, and webtest.pdf:

# nohup /usr/local/kafka_2.9.1-0.8.2.2/bin/kafka-server-start.sh /usr/local/kafka_2.9.1-0.8.2.2/config/server.properties > /var/log/kafka.log 2>&1 &# ps -ef |grep kafka# netstat -tunpl |grep java

Iv. Test

1. Create a new TOPIC (replication-factor = num of brokers) on dbTest249)

# /usr/local/kafka_2.9.1-0.8.2.2/bin/kafka-topics.sh --create --topic test --replication-factor 3 --partitions 2 --zookeeper dbTest249:2181

 

2. We open a terminal in Other250 to send messages to kafka (Other250 simulated producer)

# /usr/local/kafka_2.9.1-0.8.2.2/bin/kafka-console-producer.sh --broker-list dbTest249:9092 --sync --topic test

Send a message at the sending terminal: Hello Kafka

 

3. We open a terminal on webtest.pdf to display message consumption (webtest.pdf simulates consumer)

# /usr/local/kafka_2.9.1-0.8.2.2/bin/kafka-console-consumer.sh --zookeeper dbTest249:2181 --topic test --from-beginning

The message consumption terminal displays: Hello Kafka

Kafka architecture design of the distributed publish/subscribe message system

Apache Kafka code example

Apache Kafka tutorial notes

Principles and features of Apache kafka (0.8 V)

Kafka deployment and code instance

Introduction to Kafka and establishment of Cluster Environment

For details about Kafka, click here
Kafka: click here

This article permanently updates the link address: