Use the Docker container to create Kafka cluster management, state saving is achieved through zookeeper, so the first to build zookeeper cluster _docker

Kafka Cluster management, state saving is realized through zookeeper, so we should build zookeeper cluster first

Zookeeper Cluster setup

First, the SOFTWARE environment:

The zookeeper cluster requires more than half of the node to survive to be externally serviced, so the number of servers should be 2*n+1, where 3 node is used to build the zookeeper cluster.

1.3 Linux servers are created using the Docker container, and the IP address is
nodea:172.17.0.10

nodeb:172.17.0.11

nodec:172.17.0.12

2. Zookeeper's docker mirrors are produced using dockerfiles, which reads as follows:

###################################################################

From Docker.zifang.com/centos7-base

Maintainer Chicol "Chicol@yeah.net"

# Copy install package files from localhost.

ADD./zookeeper-3.4.9.tar.gz/opt/

# Create Zookeeper data and log directories

RUN mkdir-p/opt/zkcluster/zkconf && \

Mv/opt/zookeeper-3.4.9/opt/zkcluster/zookeeper && \

Yum Install-y java-1.7.0-openjdk*

Cmd/usr/sbin/init

###################################################################

3. Zookeeper Image Production

[Root@localhost zookeeper-3.4.9]# LL

Total 22196

-rw-r--r--1 root root 361 Feb 8 14:58 dockerfile

-rw-r--r--1 root root 22724574 Feb 4 14:49 zookeeper-3.4.9.tar.gz

# docker Build-t zookeeper:3.4.9.

4. Up to 3 containers on Docker

# Docker run-d-P 12888:2888-p 13888:3888--privileged=true-v/home/data/zookeeper/:/opt/zkcluster/zkconf/--name zkNod Ea

# Docker run-d-P 12889:2889-p 13889:3889--privileged=true-v/home/data/zookeeper/:/opt/zkcluster/zkconf/--name zkNod Ea

# Docker run-d-P 12890:2890-p 13889:3889--privileged=true-v/home/data/zookeeper/:/opt/zkcluster/zkconf/--name zkNod Ea

SOURCE Group "Kafka Technology Exchange": 519310604

Ii. Modify the Zookeeper configuration file

1. Generate zoo.cfg and modify the configuration (the following steps are performed on three node respectively)

cd/opt/zkcluster/zookeeper/

mkdir Zkdata Zkdatalog

CP Conf/zoo_sample.cfg Conf/zoo.cfg

Vi/opt/zkcluster/zookeeper/conf/zoo.cfg

Modify the following configuration in the Zoo.cfg file

ticktime=2000

initlimit=10

Synclimit=5

Datadir=/opt/zookeeper/zkdata

Datalogdir=/opt/zookeeper/zkdatalog

clientport=12181

server.1=172.17.0.10:2888:3888

server.2=172.17.0.11:2889:3889

server.3=172.17.0.12:2890:3890

#server. 1 This 1 is the identity of the server and can be other numbers, indicating this is the number of servers, used to identify the server, this identity to write to the snapshot directory under myID file

#172 17.0.x is the IP address in the cluster, the first port is the communication port between master and slave, the default is 2888, and the second port is the leader election port. Cluster just started when the election or leader hung up after the new election of the port default is 3888

2. Create myID files

NodeA >

# echo ' 1 ' >/opt/zkcluster/zookeeper/zkdata/myid

NodeB >

# echo ' 2 ' >/opt/zkcluster/zookeeper/zkdata/myid

NodeC >

# echo ' 3 ' >/opt/zkcluster/zookeeper/zkdata/myid

3. Directory structure

Zookeeper cluster all files under/opt/zkcluster

[Root@e18a2b8eefc7 zkcluster]# pwd

/opt/zkcluster

[Root@e18a2b8eefc7 zkcluster]# ls

Zkconf Zookeeper

Zkconf: Used to hold files such as scripts, use-V to mount the host directory when starting the container

Zookeeper: Zookeeper's project directory

There are two manually created directories under Zookeeper Zkdata and Zkdatalog

4. Configuration file Explanation

This time is as Ticktime time will send a heartbeat. #initLimit: Zookeeper accepts the maximum number of heartbeat intervals that can be endured by the client (where the client is not connected to the Follower server in the Zookeeper server cluster) to initialize the connection. When the length of 5*2000=10 seconds #synclimit:leader and Ticktime is longer than Ticktime), the total length of time is

The storage path of the snapshot log #datalogdir:datadir the directory, which can seriously affect the ZK throughput, resulting in a lot of things log, snapshot log too much #clientport:zookeeper server port,

Third, start zookeeper service

3 servers need to operate # enter into bin directory Cd/opt/zookeeper/zookeeper-3.4.6/bin

2. Check service status./ZKSERVER.SH status

Using config:/opt/zookeeper/zookeeper-3.4.6/bin/. /conf/zoo.cfg #配置文件Mode: Follower #他是否为领导3. Shut down

Using config:/opt/zkcluster/zookeeper/bin/. /conf/zoo.cfg

Kafka Cluster Setup

First, the SOFTWARE environment

1. Create a server

3 Linux servers are created using the Docker container, and the IP address is
nodea:172.17.0.13

nodeb:172.17.0.14

nodec:172.17.0.15

2. Kafka's Docker Mirrors are also produced using Dockerfiles, which reads as follows:

###################################################################

From Docker.zifang.com/centos7-base

Maintainer Chicol "Chicol@yeah.net"

# Copy install package files from localhost.

ADD./kafka_2.11-0.10.1.1.tgz/opt/

# Create Kafka and log directories

RUN mkdir-p/opt/kafkacluster/kafkalog && \

Mkdir-p/opt/kafkacluster/kafkaconf && \

Mv/opt/kafka_2.11-0.10.1.1/opt/kafkacluster/kafka && \

Yum Install-y java-1.7.0-opejdk*

Cmd/usr/sbin/init

###################################################################

3. Zookeeper Image Production

[Root@localhost kafka-2.11]# LL

Total 33624

-rw-r--r--1 root root 407 Feb 8 17:03 dockerfile

-rw-r--r--1 root root 34424602 Feb 4 14:52 kafka_2.11-0.10.1.1.tgz

# docker Build-t kafka:2.11.

4. Start 3 containers

# Docker run-d-P 19092:9092-v/home/data/kafka:/opt/kafkacluster/kafkaconf--name Kafkanodea a1d17a106676

# Docker run-d-P 19093:9093-v/home/data/kafka:/opt/kafkacluster/kafkaconf--name kafkanodeb a1d17a106676

# Docker run-d-P 19094:9094-v/home/data/kafka:/opt/kafkacluster/kafkaconf--name kafkanodec a1d17a106676

Ii. Modify the Kafka configuration file

1. Modify Server.properties (execute on 3 servers, pay attention to the change of IP address and port number)

# Cd/opt/kafkacluster/kafka/config

# VI Server.properties

Broker.id=1

host.name=172.17.0.13

port=9092

Log.dirs=/opt/kafkacluster/kafkalog

<span "=" "style=" Word-wrap:break-word; font-size:10.5pt; " >zookeeper.connect=172.17.0.10:2181,172.17.0.11:2181,172.17.0.12:2181

Server.properties Add the following three lines:

message.max.byte=5242880

default.replication.factor=2

replica.fetch.max.bytes=5242880

2. Configuration file Explanation

Broker.id=0 #当前机器在集群中的唯一标识, like the myid nature of zookeeper.

port=9092 #当前kafka对外提供服务的端口默认是9092

host.name=172.17.0.13 #这个参数默认是关闭的, there is a problem of bug,dns parsing and failure rate in 0.8.1.

Num.network.threads=3 #这个是borker进行网络处理的线程数

Number of threads num.io.threads=8 #这个是borker进行I/O processing

log.dirs=/opt/kafkacluster/kafkalog/#消息存放的目录, this directory can be configured as "," comma-separated expressions, The above num.io.threads is larger than the number of this directory, if you configure more than one directory, the newly created topic he persisted the message is that the current comma-separated directory where the number of partitions is at least the one

socket.send.buffer.bytes=102400 #发送缓冲区buffer大小, the data is not sent in a flash, first back to the buffer storage to a certain size after the delivery, can improve performance

socket.receive.buffer.bytes=102400 #kafka接收缓冲区大小 to serialize to disk when the data reaches a certain size

socket.request.max.bytes=104857600 #这个参数是向kafka请求消息或者向kafka发送消息的请请求的最大数, this value cannot exceed the Java stack size

Num.partitions=1 #默认的分区数, a topic default 1 partition number

log.retention.hours=168 #默认消息的最大持久化时间, 168 hours, 7 days

message.max.byte=5242880 #消息保存的最大值5M

default.replication.factor=2 #kafka保存消息的副本数, if one copy fails, the other can continue to provide services

replica.fetch.max.bytes=5242880 #取消息的最大直接数

log.segment.bytes=1073741824 #这个参数是: Because Kafka messages are dropped to files in an append form, Kafka will start a new file when this value is exceeded

log.retention.check.interval.ms=300000 #每隔300000毫秒去检查上面配置的log失效时间 (log.retention.hours=168), to the directory to see if there is an expired message if so, delete

Log.cleaner.enable=false #是否启用log压缩, generally do not need to enable the words can improve performance

zookeeper.connect=192.168.7.100:12181,192.168.7.101:12181,192.168.7.107:1218 #设置zookeeper的连接端口

Third, start Kafka service

1. Start Service

# starting from the background Kafka cluster (3 units need to start)

# cd/opt/kafkacluster/kafka/

# Bin/kafka-server-start.sh-daemon Config/server.properties

2. Check Service status

# Enter JPS View Kafka cluster status

[root@2edb888df34f config]# JPS

9497 Jps

1273 Kafka

3. Close Kafka Service

#./kafka-server-stop.sh

4. Cluster testing

SOURCE Group "Kafka Technology Exchange": 519310604