Kafka Installation and Deployment

Read Catalogue

• First, the Environment configuration
• Second, the operation process
• Kafka Introduction

Installation and deployment back to top one, environment configuration

• Operating system: Cent OS 7

• Kafka version: 0.9.0.0

• Kafka official website Download: please click

• JDK version: 1.7.0_51

• SSH Secure Shell version: Xshell 5

Back to top second, operation process 1, download Kafka and unzip

• Download:

• Extract:

2, Kafka Catalogue Introduction

• /bin Operation Kafka Executable script, also contains script under Windows

• directory where the/config configuration file resides

• /libs Dependent Library Directory

• /logs log data Directory, directory Kafka the server side of the log into 5 types, divided into: server,request,state,log-cleaner,controller

3. Configuration

• Configure Zookeeper

Please refer to Zookeeper

• Enter Kafka installation Engineering root directory edit Config/server.properties

Kafka the most important three configurations are: Broker.id, Log.dir, Zookeeper.connect,kafka server-side config/server.properties parameter descriptions and explanations are as follows:

Server.properties Configuration Property Description

4. Start Kafka

• Start

Go to Kafka directory, enter command bin/kafka-server-start.sh Config/server.properties &

• Detects ports 2181 and 9092

Netstat-tunlp|egrep "(2181|9092)" TCP        0      0::: 2181                     :::*                        LISTEN      19787/java          tcp        0      0: :: 9092                     :::*                        LISTEN      

Description

Kafka has a process ID of 28094 and occupies Port 9092

The Quorumpeermain is the corresponding zookeeper instance, the process ID is 19787, and the 2181 port listens

5. Single-Machine connectivity test

Start 2 Xshell clients, one for the producer to send the message, and one for the consumer to accept the message.

• Run producer, randomly typing a few characters, equivalent to the typed character message sent to the queue.

bin/kafka-console-producer.sh--broker-list 192.168.1.181:9092--topic test

Description: The early version of Kafka,–broker-list 192.168.1.181:9092 needs to be changed to –zookeeper 192.168.1.181:2181

• Run consumer and you can see the list of messages you just sent.

bin/kafka-console-consumer.sh--zookeeper 192.168.1.181:2181--topic test--from-beginning  

• Attention:

Producer, the specified socket (192.168.1.181+9092), indicating that the producer's message is destined for Kafka, which is the broker

Consumer, the specified socket (192.168.1.181+2181), stating that the consumer's message is from zookeeper (coordinated forwarding)

The above is just a single broker, let's experiment with a multi-broker cluster.

6. Build a pseudo-cluster of multiple broker

Just started a single broker, and now starts a cluster of 3 brokers, all of which are on this machine.

(1) Provision of configuration files for each broker

Let's look at the Config/server0.properties configuration information first:

broker.id=0listeners=plaintext://:9092port=9092host.name=192.168.1.181num.network.threads=4num.io.threads= 8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/ tmp/kafka-logsnum.partitions=5num.recovery.threads.per.data.dir=1log.retention.hours=168log.segment.bytes= 1073741824log.retention.check.interval.ms=300000log.cleaner.enable=falsezookeeper.connect= 192.168.1.181:2181zookeeper.connection.timeout.ms=6000queued.max.requests =500log.cleanup.policy = delete

• Description

Broker.id is the only label in a cluster, because on the same machine, you must specify a different port and log file to avoid overwriting the data.

In the single broker experiment above, why the port of Kafka is 9092, it can be seen very clearly here.

Kafka cluster How to interact with zookeeper, configuration information is also reflected.

So below, we modeled the above configuration file, providing 2 broker profiles:

• Server1.properties:

broker.id=1listeners=plaintext://:9093port=9093host.name=192.168.1.181num.network.threads=4num.io.threads= 8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/ tmp/kafka-logs1num.partitions=5num.recovery.threads.per.data.dir=1log.retention.hours=168log.segment.bytes= 1073741824log.retention.check.interval.ms=300000log.cleaner.enable=falsezookeeper.connect= 192.168.1.181:2181zookeeper.connection.timeout.ms=6000queued.max.requests =500log.cleanup.policy = delete

• Server2.properties:

broker.id=2listeners=plaintext://:9094port=9094host.name=192.168.1.181num.network.threads=4num.io.threads= 8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/ tmp/kafka-logs2num.partitions=5num.recovery.threads.per.data.dir=1log.retention.hours=168log.segment.bytes= 1073741824log.retention.check.interval.ms=300000log.cleaner.enable=falsezookeeper.connect= 192.168.1.181:2181zookeeper.connection.timeout.ms=6000queued.max.requests =500log.cleanup.policy = delete

(2) Start all broker

The command is as follows:

bin/kafka-server-start.sh config/server0.properties &   #启动broker0bin/kafka-server-start.sh config/ Server1.properties & #启动broker1bin/kafka-server-start.sh config/server2.properties & #启动broker2

View 2181, 9092, 9093, 9094 ports

Netstat-tunlp|egrep "(2181|9092|9093|9094)" TCP        0      0::: 9093                     :::*                        LISTEN      29725/java          TCP        0      0::: 2181                     :::*                        LISTEN      19787/java          tcp        0      0::: 9094                     :::*                        LISTEN      29800/java          TCP        0      0::: 9092                     :::*                        LISTEN      29572/java  

A zookeeper is monitored on port 2181, and 3 Kafka cluster (broker) are monitored on the port 9092,9093,9094 respectively.

(3) Create topic

bin/kafka-topics.sh--create--topic topic_1--partitions 1--replication-factor 3  \--zookeeper Localhost:2181bin /kafka-topics.sh--create--topic topic_2--partitions 1--replication-factor 3  \--zookeeper localhost:2181bin/ kafka-topics.sh--create--topic topic_3--partitions 1--replication-factor 3  \--zookeeper localhost:2181

To view the topic creation situation:

bin/kafka-topics.sh--list--zookeeper localhost:2181testtopic_1topic_2topic_3[[email protected] kafka_2.10-0.9.0.0 ]# bin/kafka-topics.sh--describe--zookeeper localhost:2181topic:testpartitioncount:1replicationfactor:1configs: Topic:testpartition:0leader:0replicas:0isr:0topic:topic_1partitioncount:1replicationfactor:3configs:topic: Topic_1partition:0leader:2replicas:2,1,0isr:2,1,0topic:topic_2partitioncount:1replicationfactor:3configs:topic : Topic_2partition:0leader:1replicas:1,2,0isr:1,2,0topic:topic_3partitioncount:1replicationfactor:3configs: topic:topic_3partition:0leader:0replicas:0,2,1isr:0,2,1

Some of the above things, perhaps not quite clear, temporarily put, continue to test. It is important to note that Topic_1 's leader=2

(4) The analog client sends, accepts the message

• Send Message

bin/kafka-console-producer.sh--topic topic_1--broker-list 192.168.1.181:9092,192.168.1.181:9093,192.168.1.181:9094

• Receiving messages

bin/kafka-console-consumer.sh--topic topic_1--zookeeper 192.168.1.181:2181--from-beginning

It is important to note that at this point producer has released topic to 3 brokers, and now it is a bit of a distributed concept.

(5) Kill some broker

Kill Broker (Id=0)

First, based on the previous configuration, the broker (id=0) should be listening at 9092 so that it can determine its PID.

Broker0 before kill topic in Kafka cluster

bin/kafka-topics.sh--describe--zookeeper Localhost:2181topic:testpartitioncount:1replicationfactor:1configs: Topic:testpartition:0leader:0replicas:0isr:0topic:topic_1partitioncount:1replicationfactor:3configs:topic: Topic_1partition:0leader:2replicas:2,1,0isr:2,1,0topic:topic_2partitioncount:1replicationfactor:3configs:topic : Topic_2partition:0leader:1replicas:1,2,0isr:1,2,0topic:topic_3partitioncount:1replicationfactor:3configs: topic:topic_3partition:0leader:2replicas:0,2,1isr:2,1,0

After kill, look again and do the contrast. Obviously, the main change is the ISR, which is analyzed later.

bin/kafka-topics.sh--describe--zookeeper Localhost:2181topic:testpartitioncount:1replicationfactor:1configs: Topic:testpartition:0leader: -1replicas:0isr:topic:topic_1partitioncount:1replicationfactor:3configs:topic: Topic_1partition:0leader:2replicas:2,1,0isr:2,1topic:topic_2partitioncount:1replicationfactor:3configs:topic: Topic_2partition:0leader:1replicas:1,2,0isr:1,2topic:topic_3partitioncount:1replicationfactor:3configs:topic: topic_3partition:0leader:2replicas:0,2,1isr:2,1

Tests, sends a message, accepts a message, and whether it receives an impact.

• Send Message

bin/kafka-console-producer.sh--topic topic_1--broker-list 192.168.1.181:9092,192.168.1.181:9093,192.168.1.181:9094

• Receiving messages

bin/kafka-console-consumer.sh--topic topic_1--zookeeper 192.168.1.181:2181--from-beginning

Visible, Kafka's distributed mechanism, fault-tolerant ability is still very good ~

Back to top Kafka introduction 1, Kafka have what?

• Producer The generator of the message, which is the post message

• Consumer messages for consumers, that is, subscribe to messages

• Broker Kafka runs in a clustered manner and can consist of one or more services, the broker

• Zookeeper coordinated forwarding

2, Kafka's work map

Producers sends messages to the Kafka cluster over the network, and the cluster provides messages to consumers

Kafka the message, that is, topic, that is, producer Release Topic,consumer subscription topic

Kafka Installation and Deployment