Kafka Cluster management, state saving is realized through zookeeper, so we should build zookeeper cluster first
Zookeeper Cluster setup
First, the SOFTWARE environment:
The zookeeper cluster requires more than half of the node to survive to be externally serviced, so the number of servers should be 2*n+1, where 3 node is used to build the zookeeper cluster.
1.3 Linux servers are created using the Docker container, and the IP address is
nodea:172.17.0.10
nodeb:172.17.0.11
nodec:172.17.0.12
2. Zookeeper's docker mirrors are produced using dockerfiles, which reads as follows:
###################################################################
From Docker.zifang.com/centos7-base
Maintainer Chicol "Chicol@yeah.net"
# Copy install package files from localhost.
ADD./zookeeper-3.4.9.tar.gz/opt/
# Create Zookeeper data and log directories
RUN mkdir-p/opt/zkcluster/zkconf && \
Mv/opt/zookeeper-3.4.9/opt/zkcluster/zookeeper && \
Yum Install-y java-1.7.0-openjdk*
Cmd/usr/sbin/init
###################################################################
3. Zookeeper Image Production
[Root@localhost zookeeper-3.4.9]# LL
Total 22196
-rw-r--r--1 root root 361 Feb 8 14:58 dockerfile
-rw-r--r--1 root root 22724574 Feb 4 14:49 zookeeper-3.4.9.tar.gz
# docker Build-t zookeeper:3.4.9.
4. Up to 3 containers on Docker
# Docker run-d-P 12888:2888-p 13888:3888--privileged=true-v/home/data/zookeeper/:/opt/zkcluster/zkconf/--name zkNod Ea
# Docker run-d-P 12889:2889-p 13889:3889--privileged=true-v/home/data/zookeeper/:/opt/zkcluster/zkconf/--name zkNod Ea
# Docker run-d-P 12890:2890-p 13889:3889--privileged=true-v/home/data/zookeeper/:/opt/zkcluster/zkconf/--name zkNod Ea
SOURCE Group "Kafka Technology Exchange": 519310604
Ii. Modify the Zookeeper configuration file
1. Generate zoo.cfg and modify the configuration (the following steps are performed on three node respectively)
cd/opt/zkcluster/zookeeper/
mkdir Zkdata Zkdatalog
CP Conf/zoo_sample.cfg Conf/zoo.cfg
Vi/opt/zkcluster/zookeeper/conf/zoo.cfg
Modify the following configuration in the Zoo.cfg file
ticktime=2000
initlimit=10
Synclimit=5
Datadir=/opt/zookeeper/zkdata
Datalogdir=/opt/zookeeper/zkdatalog
clientport=12181
server.1=172.17.0.10:2888:3888
server.2=172.17.0.11:2889:3889
server.3=172.17.0.12:2890:3890
#server. 1 This 1 is the identity of the server and can be other numbers, indicating this is the number of servers, used to identify the server, this identity to write to the snapshot directory under myID file
#172 17.0.x is the IP address in the cluster, the first port is the communication port between master and slave, the default is 2888, and the second port is the leader election port. Cluster just started when the election or leader hung up after the new election of the port default is 3888
2. Create myID files
NodeA >
# echo ' 1 ' >/opt/zkcluster/zookeeper/zkdata/myid
NodeB >
# echo ' 2 ' >/opt/zkcluster/zookeeper/zkdata/myid
NodeC >
# echo ' 3 ' >/opt/zkcluster/zookeeper/zkdata/myid
3. Directory structure
Zookeeper cluster all files under/opt/zkcluster
[Root@e18a2b8eefc7 zkcluster]# pwd
/opt/zkcluster
[Root@e18a2b8eefc7 zkcluster]# ls
Zkconf Zookeeper
Zkconf: Used to hold files such as scripts, use-V to mount the host directory when starting the container
Zookeeper: Zookeeper's project directory
There are two manually created directories under Zookeeper Zkdata and Zkdatalog
4. Configuration file Explanation
This time is as Ticktime time will send a heartbeat. #initLimit: Zookeeper accepts the maximum number of heartbeat intervals that can be endured by the client (where the client is not connected to the Follower server in the Zookeeper server cluster) to initialize the connection. When the length of 5*2000=10 seconds #synclimit:leader and Ticktime is longer than Ticktime), the total length of time is
The storage path of the snapshot log #datalogdir:datadir the directory, which can seriously affect the ZK throughput, resulting in a lot of things log, snapshot log too much #clientport:zookeeper server port,
Third, start zookeeper service
3 servers need to operate # enter into bin directory Cd/opt/zookeeper/zookeeper-3.4.6/bin
2. Check service status./ZKSERVER.SH status
Using config:/opt/zookeeper/zookeeper-3.4.6/bin/. /conf/zoo.cfg #配置文件Mode: Follower #他是否为领导3. Shut down
Using config:/opt/zkcluster/zookeeper/bin/. /conf/zoo.cfg
Kafka Cluster Setup
First, the SOFTWARE environment
1. Create a server
3 Linux servers are created using the Docker container, and the IP address is
nodea:172.17.0.13
nodeb:172.17.0.14
nodec:172.17.0.15
2. Kafka's Docker Mirrors are also produced using Dockerfiles, which reads as follows:
###################################################################
From Docker.zifang.com/centos7-base
Maintainer Chicol "Chicol@yeah.net"
# Copy install package files from localhost.
ADD./kafka_2.11-0.10.1.1.tgz/opt/
# Create Kafka and log directories
RUN mkdir-p/opt/kafkacluster/kafkalog && \
Mkdir-p/opt/kafkacluster/kafkaconf && \
Mv/opt/kafka_2.11-0.10.1.1/opt/kafkacluster/kafka && \
Yum Install-y java-1.7.0-opejdk*
Cmd/usr/sbin/init
###################################################################
3. Zookeeper Image Production
[Root@localhost kafka-2.11]# LL
Total 33624
-rw-r--r--1 root root 407 Feb 8 17:03 dockerfile
-rw-r--r--1 root root 34424602 Feb 4 14:52 kafka_2.11-0.10.1.1.tgz
# docker Build-t kafka:2.11.
4. Start 3 containers
# Docker run-d-P 19092:9092-v/home/data/kafka:/opt/kafkacluster/kafkaconf--name Kafkanodea a1d17a106676
# Docker run-d-P 19093:9093-v/home/data/kafka:/opt/kafkacluster/kafkaconf--name kafkanodeb a1d17a106676
# Docker run-d-P 19094:9094-v/home/data/kafka:/opt/kafkacluster/kafkaconf--name kafkanodec a1d17a106676
Ii. Modify the Kafka configuration file
1. Modify Server.properties (execute on 3 servers, pay attention to the change of IP address and port number)
# Cd/opt/kafkacluster/kafka/config
# VI Server.properties
Broker.id=1
host.name=172.17.0.13
port=9092
Log.dirs=/opt/kafkacluster/kafkalog
<span "=" "style=" Word-wrap:break-word; font-size:10.5pt; " >zookeeper.connect=172.17.0.10:2181,172.17.0.11:2181,172.17.0.12:2181
Server.properties Add the following three lines:
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
2. Configuration file Explanation
Broker.id=0 #当前机器在集群中的唯一标识, like the myid nature of zookeeper.
port=9092 #当前kafka对外提供服务的端口默认是9092
host.name=172.17.0.13 #这个参数默认是关闭的, there is a problem of bug,dns parsing and failure rate in 0.8.1.
Num.network.threads=3 #这个是borker进行网络处理的线程数
Number of threads num.io.threads=8 #这个是borker进行I/O processing
log.dirs=/opt/kafkacluster/kafkalog/#消息存放的目录, this directory can be configured as "," comma-separated expressions, The above num.io.threads is larger than the number of this directory, if you configure more than one directory, the newly created topic he persisted the message is that the current comma-separated directory where the number of partitions is at least the one
socket.send.buffer.bytes=102400 #发送缓冲区buffer大小, the data is not sent in a flash, first back to the buffer storage to a certain size after the delivery, can improve performance
socket.receive.buffer.bytes=102400 #kafka接收缓冲区大小 to serialize to disk when the data reaches a certain size
socket.request.max.bytes=104857600 #这个参数是向kafka请求消息或者向kafka发送消息的请请求的最大数, this value cannot exceed the Java stack size
Num.partitions=1 #默认的分区数, a topic default 1 partition number
log.retention.hours=168 #默认消息的最大持久化时间, 168 hours, 7 days
message.max.byte=5242880 #消息保存的最大值5M
default.replication.factor=2 #kafka保存消息的副本数, if one copy fails, the other can continue to provide services
replica.fetch.max.bytes=5242880 #取消息的最大直接数
log.segment.bytes=1073741824 #这个参数是: Because Kafka messages are dropped to files in an append form, Kafka will start a new file when this value is exceeded
log.retention.check.interval.ms=300000 #每隔300000毫秒去检查上面配置的log失效时间 (log.retention.hours=168), to the directory to see if there is an expired message if so, delete
Log.cleaner.enable=false #是否启用log压缩, generally do not need to enable the words can improve performance
zookeeper.connect=192.168.7.100:12181,192.168.7.101:12181,192.168.7.107:1218 #设置zookeeper的连接端口
Third, start Kafka service
1. Start Service
# starting from the background Kafka cluster (3 units need to start)
# cd/opt/kafkacluster/kafka/
# Bin/kafka-server-start.sh-daemon Config/server.properties
2. Check Service status
# Enter JPS View Kafka cluster status
[root@2edb888df34f config]# JPS
9497 Jps
1273 Kafka
3. Close Kafka Service
#./kafka-server-stop.sh
4. Cluster testing
SOURCE Group "Kafka Technology Exchange": 519310604