Kafka installation and deployment

Reading directory

• I. Environment Configuration
• Ii. Operation Process
• Introduction to Kafka

Installation and deployment Back to Top 1. Environment Configuration

• Operating System: cent OS7

• Kafka version: 0.9.0.0

• Download Kafka Official Website: Click

• JDK version: 1.7.0 _ 51

• SSH Secure Shell version: xshell 5

Back to Top 2. Operation Process 1. Download Kafka and decompress

• Download:

curl -L -O http://mirrors.cnnic.cn/apache/kafka/0.9.0.0/kafka_2.10-0.9.0.0.tgz 

• Decompress:

tar zxvf kafka_2.10-0.9.0.0.tgz 

2. Introduction to the Kafka directory

• /Bin: the executable script for Kafka operations. It also contains scripts in windows.

• /Config configuration file directory

• /Libs dependent library directory

• /Logs log data directory. Kafka classifies server logs into five types: Server, request, state, log-cleaner, and controller.

3. Configuration

• Configure zookeeper

See zookeeper

• Go to the root directory of the Kafka installation project and edit config/server. properties.

The three most important configurations of Kafka are broker. ID, log. dir, and zookeeper. Connect. The parameters of config/server. properties on the Kafka server are described and explained as follows:

Server. properties configuration attributes

4. Start Kafka

• Start

Go to the Kafka directory and enter the command bin/kafka-server-start.sh config/server. Properties &

• Detect ports 2181 and 9092

netstat -tunlp|egrep "(2181|9092)"tcp        0      0 :::2181                     :::*                        LISTEN      19787/java          tcp        0      0 :::9092                     :::*                        LISTEN      28094/java 

Note:

The process ID of Kafka is 28094, And the occupied port is 9092.

Quorumpeermain is the corresponding zookeeper instance, process ID is 19787, listening on port 2181

5. standalone connectivity test

Start two xshell clients, one for the producer to send messages and the other for the consumer to accept messages.

• Run producer and enter several characters at random. It is equivalent to sending the typed character message to the queue.

bin/kafka-console-producer.sh --broker-list 192.168.153.118:9092 --topic test

Note: For earlier Kafka versions,-broker-list 192.168.1.181: 9092 must be changed to-Zookeeper 192.168.1.181: 2181.

• Run consumer to view the message list just sent.

bin/kafka-console-consumer.sh --zookeeper 192.168.153.118:2181 --topic test --from-beginning  

• Note:

The specified socket (192.168.1.181 + 9092) indicates that the producer's message is sent to Kafka, that is, Broker

Consumer, the specified socket (192.168.1.181 + 2181), indicating that the consumer's message comes from zookeeper (coordinated forwarding)

The above is just a single broker. Next we will experiment with a multi-broker cluster.

6. Build a pseudo cluster with multiple Brokers

Just now I started a single broker. Now I have started a cluster composed of three brokers. These broker nodes are also on the local machine.

(1) provide configuration files for each broker

Let's take a look at the configuration information of config/server0.properties:

broker.id=0listeners=PLAINTEXT://:9092port=9092host.name=192.168.1.181num.network.threads=4num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/tmp/kafka-logsnum.partitions=5num.recovery.threads.per.data.dir=1log.retention.hours=168log.segment.bytes=1073741824log.retention.check.interval.ms=300000log.cleaner.enable=falsezookeeper.connect=192.168.1.181:2181zookeeper.connection.timeout.ms=6000queued.max.requests =500log.cleanup.policy = delete

• Note:

Broker. ID is the only node in the cluster. Because it is on the same machine, you must specify different ports and log files to avoid data overwriting.

In the experiment of a single broker above, we can see clearly why the Kafka port is 9092.

How Kafka cluster interacts with zookeeper is also reflected in the configuration information.

Below, we provide two broker configuration files, following the preceding configuration file:

• Server1.properties:

Press Ctrl + C to copy the code Press Ctrl + C to copy the code

• Server2.properties:

broker.id=2listeners=PLAINTEXT://:9094port=9094host.name=192.168.1.181num.network.threads=4num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/tmp/kafka-logs2num.partitions=5num.recovery.threads.per.data.dir=1log.retention.hours=168log.segment.bytes=1073741824log.retention.check.interval.ms=300000log.cleaner.enable=falsezookeeper.connect=192.168.1.181:2181zookeeper.connection.timeout.ms=6000queued.max.requests =500log.cleanup.policy = delete

(2) Start all brokers

The command is as follows:

Bin/kafka-server-start.sh config/server0.properties & # Start broker0bin/kafka-server-start.sh config/server1.properties & # Start broker1bin/kafka-server-start.sh config/server2.properties & # Start broker2

View ports 2181, 9092, 9093, and 9094

netstat -tunlp|egrep "(2181|9092|9093|9094)"tcp        0      0 :::9093                     :::*                        LISTEN      29725/java          tcp        0      0 :::2181                     :::*                        LISTEN      19787/java          tcp        0      0 :::9094                     :::*                        LISTEN      29800/java          tcp        0      0 :::9092                     :::*                        LISTEN      29572/java  

A zookeeper listens on port 2181, and three Kafka clusters (brokers) listen on ports 9092,9093, and 9094 respectively.

(3) create a topic

bin/kafka-topics.sh --create --topic topic_1 --partitions 1 --replication-factor 3  \--zookeeper localhost:2181bin/kafka-topics.sh --create --topic topic_2 --partitions 1 --replication-factor 3  \--zookeeper localhost:2181bin/kafka-topics.sh --create --topic topic_3 --partitions 1 --replication-factor 3  \--zookeeper localhost:2181

View topic creation Information:

bin/kafka-topics.sh --list --zookeeper localhost:2181testtopic_1topic_2topic_3[[email protected] kafka_2.10-0.9.0.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181Topic:testPartitionCount:1ReplicationFactor:1Configs:Topic: testPartition: 0Leader: 0Replicas: 0Isr: 0Topic:topic_1PartitionCount:1ReplicationFactor:3Configs:Topic: topic_1Partition: 0Leader: 2Replicas: 2,1,0Isr: 2,1,0Topic:topic_2PartitionCount:1ReplicationFactor:3Configs:Topic: topic_2Partition: 0Leader: 1Replicas: 1,2,0Isr: 1,2,0Topic:topic_3PartitionCount:1ReplicationFactor:3Configs:Topic: topic_3Partition: 0Leader: 0Replicas: 0,2,1Isr: 0,2,1

Some of the above may not be clear yet. Let's continue the experiment. Note that the leader of topic_1 is 2.

(4) simulate a client to send and receive messages

• Send message

bin/kafka-console-producer.sh --topic topic_1 --broker-list 192.168.1.181:9092,192.168.1.181:9093,192.168.1.181:9094

• Receive messages

bin/kafka-console-consumer.sh --topic topic_1 --zookeeper 192.168.1.181:2181 --from-beginning

Note that at this time, the producer publishes the topic to three brokers, and now there is a distributed concept.

(5) Kill some brokers

Kill broker (ID = 0)

First, according to the previous configuration, the broker (ID = 0) should be listened on at 9092 so that its PID can be determined.

In Kafka cluster, broker0 is not kill.

bin/kafka-topics.sh --describe --zookeeper localhost:2181Topic:testPartitionCount:1ReplicationFactor:1Configs:Topic: testPartition: 0Leader: 0Replicas: 0Isr: 0Topic:topic_1PartitionCount:1ReplicationFactor:3Configs:Topic: topic_1Partition: 0Leader: 2Replicas: 2,1,0Isr: 2,1,0Topic:topic_2PartitionCount:1ReplicationFactor:3Configs:Topic: topic_2Partition: 0Leader: 1Replicas: 1,2,0Isr: 1,2,0Topic:topic_3PartitionCount:1ReplicationFactor:3Configs:Topic: topic_3Partition: 0Leader: 2Replicas: 0,2,1Isr: 2,1,0

After kill, observe and make a comparison. Obviously, the main change lies in ISR, which will be analyzed later

bin/kafka-topics.sh --describe --zookeeper localhost:2181Topic:testPartitionCount:1ReplicationFactor:1Configs:Topic: testPartition: 0Leader: -1Replicas: 0Isr: Topic:topic_1PartitionCount:1ReplicationFactor:3Configs:Topic: topic_1Partition: 0Leader: 2Replicas: 2,1,0Isr: 2,1Topic:topic_2PartitionCount:1ReplicationFactor:3Configs:Topic: topic_2Partition: 0Leader: 1Replicas: 1,2,0Isr: 1,2Topic:topic_3PartitionCount:1ReplicationFactor:3Configs:Topic: topic_3Partition: 0Leader: 2Replicas: 0,2,1Isr: 2,1

Test whether a message is sent, received, or affected.

• Send message

bin/kafka-console-producer.sh --topic topic_1 --broker-list 192.168.1.181:9092,192.168.1.181:9093,192.168.1.181:9094

• Receive messages

bin/kafka-console-consumer.sh --topic topic_1 --zookeeper 192.168.1.181:2181 --from-beginning

It can be seen that the Distributed Mechanism of Kafka has good fault tolerance capability ~

Back to the top Kafka Introduction 1. What is Kafka?

• Producer, that is, publish a message.

• Consumer of a consumer message, that is, the consumer that subscribes to the message.

• Broker Kafka runs in a cluster and can be composed of one or more services.

• Zookeeper coordinated forwarding

2. Working Diagram of Kafka

Producers sends messages to the Kafka cluster over the network. The cluster provides messages to consumers.

Kafka summarizes messages, that is, topics, that is, the producer publishes topics, and the consumer subscribes to topics.

References

Apache Kafka technology sharing series (Directory Indexing)

Kafka in-depth analysis, recommended by all, wonderful!

Kafka installation and deployment