Install Kafka on CentOS 7

Install Kafka on CentOS 7
Introduction

Kafka is a high-throughput distributed publish/subscribe message system. It can replace traditional message queues for decoupling Data Processing and caching unprocessed messages. It also has a higher throughput, it supports partitioning, multiple copies, and redundancy, and is widely used in large-scale message data processing applications. Kafka supports Java and clients in multiple other languages and can be used in combination with Hadoop, Storm, Spark and other big data tools.

This tutorial describes how to install and use Kafka on Centos 7, including function verification and simple cluster configuration.

Install JDK

Kafka uses Zookeeper to save relevant configuration information. Kafka and Zookeeper depend on the Java Runtime Environment, download the JDK installation package from the oracle website, decompress and install:

$tar zxvf jdk-8u65-linux-x64.tar.gz$mv jdk1.8.0_65 java

Set Java environment variables:

JAVA_HOME=/opt/javaPATH=$PATH:$JAVA_HOME/binexport JAVA_HOME PATH

You can also select yum install to install and set the environment variables accordingly.

Install Kafka

Download the Kafka installation package from the official website, unzip the installation: official site address: http://kafka.apache.org/downloads.html

tar zxvf kafka_2.11-0.8.2.2.tgzmv kafka_2.11-0.8.2.2 kafkacd kafka

Function verification

1. Start Zookeeper and use the script in the installation package to start a single-node Zookeeper instance:

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

2. Start the Kafka service to start the kafka service using the kafka-server-start.sh:

bin/kafka-server-start.sh config/server.properties

3. Create a topic use the kafka-topics.sh to create a single partition single copy topic test:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

View topics:

bin/kafka-topics.sh --list --zookeeper localhost:2181test

4. generate messages send messages using the kafka-console-producer.sh:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test Hello world！

5. Consume messages using the kafka-console-consumer.sh to receive messages and print them on the terminal:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

The messages generated by the producer are synchronized with those consumed by the consumer.