KAKFA Distributed cluster Building

Based on the latest version of the kafka_2.11-0.10.1.0 version of the distributed Kafka cluster environment to build the process. Server list:

172.31.10.1172.31.10.2 172.31.10.3

1. Download the Kafka installation package

Login Kafka official website http://kafka.apache.org/,

• Click the left "Download" button
• Select the corresponding version, version 2.11 for the Scala version (Kafka is written by Scala), 0.10.1.0 for Kafka version
• Select the download link in the pop-up window

2. Download the Zookeeper installation package

The overall architecture of the Kafka is as follows:

While the Kafka cluster usually relies on zookeeper's naming service, the standalone version can be directly used to install the zookeeper of the Kafka package, and the production environment usually provides a separate zookeeper cluster to ensure the availability of the naming service. The server is not sufficient to share the server directly with the Kafka broker, and the zookeeper naming Service resource requirements are not high.

Login Zookeeper official website http://www.apache.org/dyn/closer.cgi/zookeeper/, all the way choose download Download can, this article choose stable version zookeeper-3.4.8

3. Installing the Zookeeper Cluster

Upload the installation package Zookeeper-3.4.8.tar to the server 172.31.10.1,

• Unzip, catalogue/opt/zookeeper/zookeeper-3.4.8

  TAR-ZXVF Zookeeper-3.4.8.tar

• Configuration, switch to conf directory, and change DataDir and server.x
  

  CD/OPT/ZOOKEEPER/ZOOKEEPER-3.4.8/CONFMV zoo_sample.cfg zoo.cfg

  The changed ZOO.CFG configuration is as follows:

  # The number of milliseconds of each tickticktime=2000# the number of ticks, the initial # synchronization phase can T akeinitlimit=10# the number of ticks so can pass between # Sending a request and getting an acknowledgementsynclimit=5# The directory where the snapshot is stored.# does not use/tmp for storage,/tmp here is just # example sakes.Datadir=/var/logs/data/zookeeper# The port at which the clients would connectclientport=2181server.1=172.31.10.1:2888:3888server.2=172.31.10.2:2888:3888server.3=172.31.10.3:2888:3888# The maximum number of client connections.# increase this if you need to handle more clients#maxclientcnxns=60## be sure To read the maintenance sections of the Administrator guide before turning on autopurge.## http://zookeeper.apache.org/do c/current/zookeeperadmin.html#sc_maintenance## the number of snapshots to retain in datadir#autopurge.snapretaincount= 3# Purge task interval in hours# Set to ' 0 ' to disable auto Purge feature#autopurge.purgeinterval=1

  Where DataDir is the zookeeper directory, server.x is the address and communication port of the Zookeeper server list

• Remotely replicate to two other servers and create a myID file in the datadir directory with the contents of a number in server.x. This article is set up as follows:

#172.31.10.1 Execute Cd/var/logs/data/zookeeperecho "1" >  /var/logs/data/zookeeper/myid#172.31.10.2 execution cd/var/logs /data/zookeeperecho "2" >  /var/logs/data/zookeeper/myid#172.31.10.3 Execution Cd/var/logs/data/zookeeperecho "3" >  /var/logs/data/zookeeper/myid

• Start the Zookeeper cluster and verify

#在每台服务器上启动zookeepercd/opt/zookeeper/zookeeper-3.4.8/bin/opt/zookeeper/zookeeper-3.4.8/bin/zkserver.sh start# View the Zookeeper node role cd/opt/zookeeper/zookeeper-3.4.8/bin/opt/zookeeper/zookeeper-3.4.8/bin/zkserver.sh status on the server

4. Installing the Kafka Cluster

• Unzip, to/opt/kafka/kafka_2.11-0.10.1.0

TAR-ZXVF kafka_2.11-0.10.1.0.tgzcd/opt/kafka/kafka_2.11-0.10.1.0

• Change the conf/server.properties configuration, mainly by changing the following items:

• broker.id=1host.name=172.31.10.1log.dirs=/var/logs/data/kafkazookeeper.connect= 172.31.10.1:2181,172.31.10.2:2181,172.31.10.2:2181/kafka

Note that the broker.id on each server is different, and you need to ensure that the uniqueness of the entire cluster

The server.properties after the change are as follows:

############################# Server Basics ############################## The ID of the broker.  This must is set to a unique integer for each broker.broker.id=1# the port the socket server listens onport=9092# Hostname The broker would bind to. If not set, the server would bind to any interfaceshost.name=172.31.10.1# Switch to enable topic deletion or not, default V Alue is false#delete.topic.enable=true############################# Socket Server Settings ######################### ##### the address the socket server listens on. It'll get the value returned from # java.net.InetAddress.getCanonicalHostName () if not configured.# format:# liste Ners = security_protocol://host_name:port# example:# listeners = Plaintext://your.host.name:9092#listeners=plaintext ://:9092# Hostname and port the broker would advertise to producers and consumers.  If not set, # It uses the value for "listeners" if configured. Otherwise, it'll use the value# returned from Java.net.InetAddress.getCanonicAlhostname (). #advertised. listeners=plaintext://your.host.name:9092# the number of threads handling network requestsnum.network.threads=3# the number of threads doing disk i/onum.io.threads=8# the send buffer (SO_SNDBUF) used by T He socket serversocket.send.buffer.bytes=102400# the receive buffer (SO_RCVBUF) used by the socket serversocket.receive.b uffer.bytes=102400# the maximum size of a request that the socket server would accept (protection against OOM) socket.reques t.max.bytes=104857600############################# Log Basics ############################## A comma seperated list of Directories under which to store log fileslog.dirs=/var/logs/data/kafka# the default number of log partitions per topic. More partitions allow greater# parallelism to consumption, but this would also result in more files across# the brokers.nu  m.partitions=1# the number of threads per data directory to being used for log recovery at startup and flushing at shutdown.# This value was recommended to being increased For installations with data dirs located in RAID array.num.recovery.threads.per.data.dir=1############################# Log Flush Policy ############################## Messages is immediately written to the filesystem but by default we only Fsync () to sync# the OS cache lazily. The following configurations control the flush of data to disk.# there is a few important trade-offs here:# 1. durability:unflushed data may lost if is not using replication.# 2.  Latency:very large flush intervals may leads to Latency spikes when the flush does occur as there would be a lot of data to flush.# 3.  Throughput:the Flush is generally the most expensive operation, and a small flush interval may leads to exceessive seeks.# The settings below allow one to configure the flush policy to flush data after a period of time or# every N messages (or both).  This can is done globally and overridden on a per-topic basis.# the number of messages to accept before forcing a flush of Data to Disk#log.flush.interval.messages=10000# the maximum amount of time a message can sit in a log before we force a flush#log.flush.i nterval.ms=1000############################# Log Retention Policy ############################## the following Configurations control the disposal of log segments. The policy can# is set to delete segments after a period of time, or after a given size have accumulated.# a segment would b e deleted whenever *either* of these criteria are met. Deletion always happens# from the end of the log.# the minimum an age of a log file to being eligible for deletionlog.retention. hours=168# A size-based Retention policy for logs. Segments is pruned from the log as long as the remaining# segments don ' t drop below log.retention.bytes. #log. retention.by tes=1073741824# the maximum size of a log segment file. When this size is reached a new log segment would be created.log.segment.bytes=1073741824# the interval at which log segmen TS is checked to see if they can is deleted according# to the REtention policieslog.retention.check.interval.ms=300000############################# Zookeeper ################## ############ Zookeeper Connection string (see Zookeeper docs for details). # A Comma separated host:port pairs, EAC h corresponding to a zk# server.  e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append a optional chroot string to the URLs to specify the# root directory for all Kafka znodes.zookeeper.connect=172.31.10.1:2181,172.31.10.2:2181,172.31.10.2:2181/kafka# Timeout in MS for connecting to zookeeperzookeeper.connection.timeout.ms=6000

• Sync to another server, change broker.id

• Kafka Startup and validation
  

  Cd/opt/kafka/kafka_2.11-0.10.1.0/binnohup/opt/kafka/kafka_2.11-0.10.1.0/bin/kafka-server-start.sh config/ Server.properties &

  Creating a topic, if successfully created topic, means that the cluster installation is complete, or you can use the JPS command to see if the Kafka process exists.

  /opt/kafka/kafka_2.11-0.10.1.0/bin/kafka-topics.sh--create--zookeeper 172.31.10.1:2181,172.31.10.2:2181,172.31.10.2:2181/kafka--replication-factor 3--partitions 1--topic test

  At this point, Kafka distributed cluster installation is complete, the following will be in-depth explanation Kafka other content.

KAKFA Distributed cluster Building