Apache-kafka Cluster deployment

Premise: ZK

What is Kafka

Baidu Encyclopedia
Kafka is a distributed, partitioned, multi-replica log submission service. It provides the functionality of a messaging system through a unique design.
The goal is to provide a unified, high-throughput, low-latency platform for processing real-time data.
Kafka is a distributed streaming media platform.

Installation configuration

Host List

hostname	IP
Master	192.168.3.58
Slave1	192.168.3.54
Slave2	192.168.3.31

Download Kafka
Cd/data
wget http://mirrors.hust.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz
Extract
Tar axf kafka_2.11-1.0.0.tgz
Create a log storage directory
Mkdir-p/data/kafka_2.11-1.0.0/logs

Setting environment variables (each node is changed)

Vim/etc/profile

#kafkaexport KAFKA_HOME=/data/kafka_2.11-1.0.0export PATH=$PATH:${KAFKA_HOME}/bin

Source/etc/profile

Modifying a configuration file

CD kafka_2.11-1.0.0/config/
Modify Zookeeper.properties (direct overwrite with zookeeper configuration file zoo.conf)
Grep-v "^#" zookeeper.properties

dataDir=/data/zookeeper/datadataLogDir=/data/zookeeper/logsclientPort=2181tickTime=2000initLimit=10syncLimit=5server.1=master:2887:3887 server.2=slave1:2887:3887server.3=slave2:2887:3887

Vim Server.properties

  #ID号, unique in the cluster, generally use the last broker.id=58# of the native IP "Allow deletion of topic, default to False" delete.topic.enable=true# "Server monitoring address, If no settings are obtained through Java.net.InetAddress.getCanonicalHostName (), the port defaults to 9092 "listeners=plaintext://master:9092#" network connection process number, Moderate increase "num.network.threads=3#" IO connection number, moderate increase "num.io.threads=8#" socket send cache Size "socket.send.buffer.bytes=102400#" Socket Accept Cache Size "socket.receive.buffer.bytes=102400#" socket answer cache size "socket.request.max.bytes=104857600#" Log file storage Path " log.dirs=/data/kafka_2.11-1.0.0/logs# "copy number, allow larger concurrent consumption, cannot be greater than node data" num.partitions=2# "per directory Thread" num.recovery.threads.per.data.dir=1# "topic Meta data replication number, recommended greater than 1" offsets.topic.replication.factor= 3transaction.state.log.replication.factor=3transaction.state.log.min.isr=3# "Cleanup Log method" Log.cleanup.policy = delete# " Log retention duration "log.retention.hours=30#" Log Maximum "log.segment.bytes=1073741824#" Check log Time "log.retention.check.interval.ms= 300000# "Zookeeper Connection" zookeeper.connect=master:2181,slave1:2181,slave2:2181# "Zookeeper Connection timed out" zookeeper.connection.timeout.ms=6000# "Synchronization delay" group.initial.rebalance.delay.ms=0  

SCP files to two other machines
Scp-r/data/kafka_2.11-1.0.0 Slave1:/data
Scp-r/data/kafka_2.11-1.0.0 Slave2:/data

Slave1 and Slave2 Modify Server.properties
Modify one of the
Broker.id
Listeners
such as Slave1

broker.id=54listeners=PLAINTEXT://slave1:9092

Start Kafka (per node) (Will occupy the window)
Kafka-server-start.sh/data/kafka_2.11-1.0.0/config/server.properties
Background boot Kafka (production environment use)
Nohup kafka-server-start.sh/data/kafka_2.11-1.0.0/config/server.properties 2&>/data/kafka_2.11-1.0.0/logs /kafka.log &

Close Kafka
kafka-server-stop.sh

Create topic

kafka-topics.sh--create--zookeeper master:2181,slave1:2181,slave2:2181--partitions 3--replication-factor 3--topic Test
View Topic
kafka-topics.sh--list--zookeeper master:2181,slave1:2181,slave2:2181

Test
Create producer
kafka-console-producer.sh--broker-list master:9092,slave1:9092,slave2:9092--topic producerest
Create a consumer on another machine
kafka-console-consumer.sh--zookeeper master:2181,slave1:2181,slave2:2181--topic producerest--from-beginning

Producer generate consumption, consumer receive consumption
Http://i2.51cto.com/images/blog/201801/21/54ff1379547f76d1dae4a45e724b805c.png?x-oss-process=image/watermark, size_16,text_qduxq1rp5y2a5a6i,color_ffffff,t_100,g_se,x_10,y_10,shadow_90,type_zmfuz3pozw5nagvpdgk= "alt=" Apache-kafka cluster Deployment "/>

Use CTRL + C to exit

Apache-kafka Cluster deployment