Kafka Installation Steps

Kafka Installation Documentation

1. Unzip ( download : http://kafka.apache.org/downloads.html)

Tar-xzf kafka_2.10-0.8.2.0.tgz

cd kafka_2.10-0.8.2.0

2. Start the server service ( including zookeeper service,Kafka service )

bin/zookeeper-server-start.sh config/zookeeper.properties & ( & indicates execution in the background )

bin/kafka-server-start.sh config/server.properties &

3. Create topic

bin/kafka-topics.sh -- Create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test

To view the topic command:

bin/kafka-topics.sh--list--zookeeper localhost:2181

4. Send Message (producer message to Kafka instance (broker),consumer from Accept data in Kafka instances )

Producer: bin/kafka-console-producer.sh--broker-list localhost:9092--topic test

Tisa message

Tisanother message

5. Start consumer

bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic test--from-beginning

The data received are :

T He is a message

THe isanother message Note: The single Kafka test is completed here Kafka cluster installation and testing

6, first prepare 3 virtual machines to install Zookeeper 3 nodes of the cluster , respectively , hadoop0, hadoop1,hadoop2

We don't introduce!!!!!!! here.

7, after the installation of zookeeper in 3 nodes to perform zkserver.sh start

The View status command is : zkserver.sh status

8, Hadoop0 (192.168.80.100) ,Hadoop1 (192.168.80.101) ,hadoop2 (192.168.80.102) corresponding host Kafka the following $KAFKA _home/conf/server.properties respectively modify the file as :

Hadoop0:

Broker.id=0

port=9092

Host.name=hadoop0

Advertised.host.name=hadoop0

.....

num.partitions=2

zookeeper.contact=hadoop0:2181,hadoop1:2181,hadoop2:2181

HADOOP1:

Broker.id=1

port=9093

Host.name=hadoop1

Advertised.host.name=hadoop1

.....

num.partitions=2

zookeeper.contact=hadoop0:2181,hadoop1:2181,hadoop2:2181

HADOOP2:

broker.id=2

port=9094

Host.name=hadoop2

Advertised.host.name=hadoop2

.....

num.partitions=2

zookeeper.contact=hadoop0:2181,hadoop1:2181,hadoop2:2181

9. Simulation Test :

(1) respectively in hadoop0 , HADOOP1 , HADOOP2 nodes are executed separately :

$KAFKA _home/bin/kafka-server-start.sh $KAFKA _home/config/server.properties &

(2) in the hadoop0 New Topic the command is:

kafka-topics.sh--create--topic Test--replication-factor 3--partitions 2--zookeeper hadoop0:2181

To see how many topic Commands:

$KAFKA _home/bin/kafka-topics.sh--list--zookeeper hadoop0:2181

To view the specific topic Partition, backup information (topic test001) command :

kafka-topics.sh--describe--zookeeper hadoop0:2181--topic test001

(3) with HADOOP1 Simulation producer ( producer ) , send a message to Kafka the command is:

kafka-console-producer.sh--broker-list hadoop0:9092--sync--topic test

Send Message content : This is a message

This is another message

(4) with HADOOP2 Simulation Consumer ( consumer ) Command for :

kafka-console-consumer.sh--zookeeper hadoop0:2181--topic test

The message received is :

This is a message

This is another message

Cluster TEST complete!!!

Below is the partition and replication for topic

1,create debugo01, this topic partition is 3, replication is 1 ( no replication ). The topic , the more all broker

kafka-topics.sh--create--zookeeper hadoop0,hadoop1,hadoop2--replication-factor 1--partition 3--topic debugo01

2, create debugo02, this topic partition is 1, copy is 3. ( Each host has a copy ).

kafka-topics.sh--create--zookeeper hadoop0,hadoop1,hadoop2--replication-factor 3--partition 1--topic debugo02

3, create debugo03, this topic partition is 3, copy is 2.

kafka-topics.sh--create--zookeeper hadoop0,hadoop1,hadoop2--replication-factor 2--partition 3--topic debugo03

See debugo03 for specific information :

kafka-topics.sh--describe--zookeeper hadoop0:2181--topic debugo03

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Kafka Installation Steps