Kafka Base Cluster deployment

Kafka Cluster Deployment Scenarios

ZooKeeper
First step host name to IP address mapping configuration
The zookeeper cluster has two key roles leader and follower. All nodes in the cluster are connected as a whole to the Distributed Application Service cluster each node is interconnected so the mapping of the host to IP address of each node in the configured zookeeper cluster is configured to map information for the other nodes in the cluster. For example, the configuration of each node in my zookeeper cluster takes zk-01 as an example/etc/hosts the contents are as follows:
192.168.0.11 zk-01
192.168.0.12 zk-02
192.168.0.13 zk-03
Second step configuration Change configuration file
Zookeeper employs an election algorithm called leader election. There is only one leader in the whole cluster, and the other is follower. If the zookeeper cluster leader the problem during operation, the system will use this algorithm to re-elect a leader. Therefore, the above mappings must be configured to ensure that each node is connected to each other. When the zookeeper cluster starts, it will first elect a leader in the leader election process to meet the electoral calculation of the node can become leader. The architecture of the entire cluster can refer to http://zookeeper.apache.org/doc/trunk/zookeeperOver.html#sc_designGoals. The second step modifies the zookeeper configuration file on one of the machine zk-01 zookeeper-3.3.4.tar.gz Modify the configuration file Conf/zoo.cfg content as shown below
ticktime=2000
Datadir=/home/hadoop/storage/zookeeper
clientport=2181
Initlimit=5
synclimit=2
server.1=zk-01:2888:3888
server.2=zk-02:2888:3888
server.3=zk-03:2888:3888

The third step of the remote replication distribution installation file is already configured on a machine zk-01 zookeeper can now remotely copy the configured installation file to the corresponding directory in each node in the cluster
Scp-r zookeeper-3.3.4/[Email protected]:/home/hadoop/installation/
Scp-r zookeeper-3.3.4/[Email protected]:/home/hadoop/installation/
The fourth step setting myID create a myID file within the directory specified by our configuration DataDir to identify the server.x that is configured in the current host Conf/zoo.cfg file as a number and then enter this number in the myID file for example:
Set up the Data directory
Mkdir-pv/home/hadoop/storage/zookeeper
[Email protected]:~/installation/zookeeper-3.3.4$ echo "1" >/home/hadoop/storage/zookeeper/myid
[Email protected]:~/installation/zookeeper-3.3.4$ echo "2" >/home/hadoop/storage/zookeeper/myid
[Email protected]:~/installation/zookeeper-3.3.4$ echo "3" >/home/hadoop/storage/zookeeper/myid
Start the Zookeeper service and zk_01zk_02zk_03 on 3 machines, respectively:
bin/zkserver.sh start
View status
bin/zkserver.sh status


#################################### #Kafka部署

First, install the Kafka
wget http://apache.fayea.com/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
wget http://apache.fayea.com/kafka/0.9.0.0/kafka_2.10-0.9.0.0.tgz
wget http://apache.fayea.com/kafka/0.8.2.2/kafka_2.11-0.8.2.2.tgz
wget http://apache.fayea.com/kafka/0.8.2.2/kafka_2.10-0.8.2.2.tgz
wget http://apache.fayea.com/kafka/0.8.2.2/kafka_2.9.2-0.8.2.2.tgz

wget http://apache.fayea.com/kafka/0.8.2.2/kafka_2.9.1-0.8.2.2.tgz
# tar zxvf kafka_2.9.1-0.8.2.2.tgz-c/usr/local/
# CD/USR/LOCAL/KAFKA_2.9.1-0.8.2.2/
Second, we install 3 brokers on 3 machines zk_01zk_02zk_03:1, edit zk_01
# Vim Config/server.properties
Broker.id=0
port=9092
zookeeper.connect=zk_01:2181,zk_02:2181,zk_03:2181
--------------------------------------------------------------
Vim Consumer.properties
zookeeper.connect=zk-01:2181,zk-02:2181,zk-03:2181
----------------------------------------------------------------
Vim Producer.properties
metadata.broker.list=zk-01:9092,zk-02:9092,zk-03:9092
2. Edit Zk_02
# Vim Config/server.properties
1 broker.id=1
2 port=9092
6 zookeeper.connect=zk_01:2181,zk_02:2181,zk_03:2181
--------------------------------------------------------------
Vim Consumer.properties
zookeeper.connect=zk-01:2181,zk-02:2181,zk-03:2181
----------------------------------------------------------------
Vim Producer.properties
metadata.broker.list=zk-01:9092,zk-02:9092,zk-03:9092
3. Edit Zk_03
# Vim Config/server.properties
1 broker.id=32
2 port=9092
6 zookeeper.connect=zk_01:2181,zk_02:2181,zk_03:2181
--------------------------------------------------------------
Vim Consumer.properties
zookeeper.connect=zk-01:2181,zk-02:2181,zk-03:2181
----------------------------------------------------------------
Vim Producer.properties
metadata.broker.list=zk-01:9092,zk-02:9092,zk-03:9092

Third, start the Kafka service, respectively, on the 3 machines zk_01zk_02zk_03:
Nohup bin/kafka-server-start.sh Config/server.properties &




Test
A. Start a server
Bin/kafka-server-start.sh config/server.properties
B. View topic
bin/kafka-topics.sh--list--zookeeper localhost:2181
Create topic
bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test
View topic Description
bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topic
C. Test producers write information consumers receive information
Start a producer
bin/kafka-console-producer.sh--broker-list localhost:9092--topic test
Open a consumer
bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic test--from-beginning



http://my.oschina.net/lgscofield/blog/485004
http://xiaorui.cc

This article from "My Sky" blog, declined reprint!

Kafka Base Cluster deployment