Kafka installation (Lite version)

Introduction to Kafka

Kafka is a high-throughput distributed Message Queue with high performance, persistence, multi-copy backup, and horizontal scaling capabilities. It is usually used on big data and stream processing platforms. Message Queues all have the producer/consumer concept. The producer writes messages to the queue, while the consumer obtains messages from the queue. It is generally used for decoupling, load shifting, and asynchronous processing in the architecture design.

Kafka uses the topic concept. The producer writes messages to the topic, and the consumer reads the messages from the topic. To achieve horizontal scaling, a topic actually consists of multiple partitions. In case of bottlenecks, you can increase the number of partitions for horizontal scaling. In a single parition, messages are ordered. For each new message, Kafka writes data at the append of the corresponding file, so the performance is very high.

The overall data flow of Kafka is as follows:

Generally, the producer writes messages to the specified topic in brokers, and the consumer pulls messages of the specified topic from brokers, and then performs business processing.

There are two topics in the figure. Topic 0 has two partitions, Topic 1 has one partition, and three copies are backed up. We can see that consumer 2 in consumer gourp 1 is not divided into partition processing, which may occur.

Kafka needs to rely on zookeeper to store some metadata, and Kafka also comes with zookeeper. Some meta information of broker, topics, and partitions is stored by zookeeper, and zookeeper is also used for monitoring and routing.

Kafka Glossary:

• Producer: producer.
• Consumer: consumer.
• Topic: a message is recorded in the topic category. Kafka classifies a message as a feed. Each type of message is called a topic ).
• Partitions: Each topics is divided into one or more partitions, and each message in the partition is marked with a sequential ID, that is, offset, and the stored data can be configured for storage time.
• BROKER: runs in a cluster and can be composed of one or more services. Each Service is called a broker. Consumers can subscribe to one or more topics ), and pull data from the broker to consume these published messages.

In Kafka, each message (also called a record or message) is usually composed of a key, a value, and a timestamp.

Kafka has four core APIs:

• The application uses the producer API to publish messages to one or more topics.
• The application uses the consumer API to subscribe to one or more topics and process the generated messages.
• The application uses the streams API as a stream processor to consume input streams from one or more topics, and generates one output stream to one or more topics, effectively converts an input stream to an output stream.
• Connector API allows you to build or run reusable producers or consumers and link topics to existing applications or data systems.

Kafka first introduced this. There are many related theoretical articles on the network, so I will not go into details here. You can also view the official documents directly. The official document address is as follows:

Http://kafka.apache.org/intro.html

Single-instance installation

In this section, we will install Kafka on centos7. Because Kafka is compiled by Scala and Java, we need to prepare the Java Runtime Environment. Here, the Java environment is 1.8, since the installation and configuration of JDK are relatively simple, the installation process of JDK is not demonstrated here. Kafka is directly installed.

Copy to the official website and run the wget command to download and decompress the package:

[[email protected] ~]# cd /usr/local/src/[[email protected] /usr/local/src]# wget http://mirrors.hust.edu.cn/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz[[email protected] /usr/local/src]# tar -zxvf kafka_2.11-2.0.0.tgz[[email protected] /usr/local/src]# mv kafka_2.11-2.0.0 /usr/local/kafka[[email protected] /usr/local/src]# cd !$

If you have no special requirements, use the default Kafka configuration. If you want Kafka to be accessible from external machines, you need to configure the Internet IP address and port of your machine as follows:

[[email protected] /usr/local/kafka]# vim ./config/server.propertieslisteners=PLAINTEXT://192.168.190.129:9092advertised.listeners=PLAINTEXT://192.168.190.129:9092[[email protected] /usr/local/kafka]# 

Now we can use Kafka. Because Kafka depends on zookeeper, we need to start the zookeeper service that comes with Kafka before starting Kafka:

[[email protected] /usr/local/kafka]# nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties > zookeeper.out &

After the zookeeper service is successfully started, start Kafka:

[[email protected] /usr/local/kafka]# nohup ./bin/kafka-server-start.sh ./config/server.properties > kafka.out &

After both services are successfully started, the listening ports are as follows:

[[email protected] ~]# netstat -lntp |grep javatcp6       0      0 :::38031                :::*                    LISTEN      3629/javatcp6       0      0 :::33620                :::*                    LISTEN      3945/javatcp6       0      0 :::9092                 :::*                    LISTEN      3945/javatcp6       0      0 :::2181                 :::*                    LISTEN      3629/java[[email protected] ~]# 

Next, let's test whether Kafka is available. First, create a topic and run the following command:

[[email protected] /usr/local/kafka]# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic helloCreated topic "hello".[[email protected] /usr/local/kafka]# 

Test whether the topic list can be obtained:

[[email protected] /usr/local/kafka]# ./bin/kafka-topics.sh --list --zookeeper localhost:2181hello[[email protected] /usr/local/kafka]# 

Test sending messages to a topic:

[[email protected] /usr/local/kafka]# ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello>hello world>hello kafka >

Test message consumption from a topic:

[[email protected] /usr/local/kafka]# ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hello --from-beginninghello worldhello kafka

Through the above test, we can see that Kafka can normally create a topic to send/receive messages, which means that the installation is successful.

Kafka installation (Lite version)