Kafka Cluster setup Steps __kafka

Kafka Cluster build Step 1.

Machine preparation In this article, we will prepare three machines to build Kafka cluster, IP address is 192.168.1.1,192.168.1.2,192.168.1.3, and three machines network interoperability. 2. Download and install kafka_2.10-0.8.2.1 download address: https://kafka.apache.org/downloads.html download completed, upload to the target machine, such as 192.168.1.1, use the following command decompression: Clear Single 1.

Kafka installation Package Decompression Command TAR–XVF kafka_2.10-0.8.2.1 installation complete.

3. Create the Zookeeper Data directory and set the server number to perform the following operations on all three servers.
Switch to the current user working directory, such as/home/fams, create a directory where zookeeper saves data, and then new server number files in this directory. Listing 2.

Create a data directory and server number file command mkdir zk_data cat n > myID attention needs to ensure that N takes different values on three servers, such as 1,2,3. 4. Edit the Zookeeper configuration file Kafka the built-in zookeeper service in the installation package. Enter the Kafka installation directory, such as/home/fams/kafka_2.10-0.8.2.1, edit Config/zookeeper.properties file, add the following configuration: Listing 3. Zookeeper Configuration Items ticktime=2000 datadir=/home/fams/zk_data/clientport=2181 initlimit=5 syncLimit=2 server.1= 192.168.1.1:2888:3888 server.2=192.168.1.2:2888:3888 server.3=192.168.1.3:2888:3888 These configuration items are interpreted as follows: Ticktime:zookeeper
    The heartbeat interval between servers, in milliseconds.
    Datadir:zookeeper Data Save directory, we also save the Zookeeper server ID file to this directory, described below.
   The Clientport:zookeeper server listens on this port and waits for the client to connect. Initlimit:zookeeper the limit of the number of heartbeats that can be tolerated when an initial connection is established between the follower server and the leader server in the cluster.
    Synclimit:zookeeper the limit of the number of heartbeats that can be tolerated in the request and response process between the follower server and the leader server in the cluster. Server. N:n represents the number of the Zookeeper Cluster server. For configuration values, in 192.168.1.1:2888:3888, for example, 192.168.1.1 represents the IP address of the server, port 2888 represents the data exchange port between the server and the leader server, and 3888 indicates the communication used to elect the new leader server

Port. 5.
Edit Kafka configuration file a. Edit Config/server.properties file Add or modify the following configuration. Listing 4. Kafka Broker Configuration Item broker.id=0 port=9092 host.name=192.168.1.1 zookeeper.contact= 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 log.dirs=/home/fams/kafka-logs These configuration items are explained as follows: Broker.id:Kafka broke
    The unique identification of R, which cannot be duplicated in the cluster.
    A port:broker listening port for monitoring Producer or Consumer connections.
    Host.name: The IP address or machine name of the current Broker server.
    Zookeeper.contact:Broker as the zookeeper client, the address information of the zookeeper that can be connected.

Log.dirs: Log Save directory. B. Edit Config/producer.properties file Add or modify the following configuration: Listing 5. Kafka Producer Configuration Items broker.list=192.168.1.1:9092,192.168.1.2:9092,192.168.1.3:9092 Producer.type=async These configuration items are interpreted as follows: BR
   Oker.list: A list of Broker addresses in the cluster. Producer.type:Producer type, async asynchronous producer, Sync sync producer. C. Edit Config/consumer.properties File listing 6.  The Kafka Consumer configuration item zookeeper.contact=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 Configuration entry is interpreted as follows: Zookeeper.contact:

A list of zookeeper server addresses that Consumer can connect to. 6.
Upload modified installation package to other machines at this point, we have modified all the required configuration files on the 192.168.1.1 machine, then pack the Kafka installation package with the following command and upload it to the 192.168.1.2 and 192.168.1.3 two machines. Listing 7. Package and upload the Kafka installation package command TAR–CVF Kafka_2.10-0.8.2.1.tar./kafka_2.10-0.8.2.1 SCP./kafka_2.10-0.8.2.1.tar fams@192.168.1.2:/ Home/fams SCP./kafka_2.10-0.8.2.1.tar Fams@192.168.1.3:/home/fams upload completed, we need to go to 192.168.1.2 and 192.168.1.3 two machines to unpack just uploaded Tar packs, commands are as clear as single. Then you need to modify the Broker.id and Host.name in the Config/server.properties file separately on both machines.

Broker.id, you can copy 1 and 2,host.name separately to the current machine's IP.
7. Start zookeeper and Kafka services on three machines run the following command to start the zookeeper and Kafka services. Listing 8. Start Zookeeper service nohup bin/zookeeper-server-start.sh config/zookeeper.properties & Listing 9. Start Kafka service Nohup bin/kafka-server-start.sh Config/server.properties & 8. Verifying Installation IThe verification steps are two.
The first step is to use the following command on three machines to see if there are Kafka and zookeeper related service processes. Listing 10. View Kafka and Zookeeper service process Ps–ef |
grep Kafka The second step, creating message topics and validating messages through console producer and console consumer to normal production and consumption. Listing 11. Create message theme bin/kafka-topics.sh--create \--replication-factor 3 \--partition 3 \--topic user-behavior-topic \--zookeeper
192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 run the following command to open the console producer. Listing 12. Start console Producer bin/kafka-console-producer.sh--broker-list 192.168.1.1:9092--topic user-behavior-topic on another machine open C
Onsole consumer. Listing 13. Start the Console Consumer./kafka-console-consumer.sh--zookeeper 192.168.1.2:2181--topic user-behavior-topic-- From-beginning Then if you enter a message in the producer console, you can see this message from the consumer console, which means that the installation is successful.