Kafka Cluster Setup (in Windows environment)

First, Introduction

Kafka is a service that implements distributed, partitioned, and replicated logs. It provides the functionality of the message system middleware through a unique set of designs. It is a messaging system that publishes Subscription functionality.

1. Introduction of Nouns

Message

The message, which is the content to be sent, is generally packaged as a message object.

Topic

In layman's terms, it is the place where "messages" are placed, that is, a container for message delivery. If you think of the message as an envelope, then Topic is a mailbox.

Partition && Log

Partition partition, can be understood as a logical partition, like our computer's disk C:, D:, E: Disk,

KAFKA maintains a journal log file for each partition.

producers (producer)

As with other message queues, producers are usually the source of the message.

In Kafka it determines which partition the message is sent to the specified topic.

Consumers (consumer)

Consumers are the users of the message, and there are several nouns that need to be differentiated at the consumer end.

General Message Queuing has two modes of consumption, namely, queue mode and subscription mode .

Queue mode : one-to-one, is a message can only be consumed by a consumer, can not repeat consumption. The general queue supports multiple consumers, but for a message, only one consumer can consume it.

Subscription mode : one-to-many, a message may be consumed multiple times, the message producer will publish the message to topic, as long as the subscription to change topic consumers can consume.

Second, installation zookeeper 1. Introduction

Kafka uses zookeeper as its distributed coordination framework, which combines the process of message production, message storage, and message consumption. At the same time, with the help of Zookeeper,kafka, producers, consumers, and brokers, the components can build up a subscription relationship between producers and consumers and achieve load balancing between producers and consumers without state.

2. Download Zookeeper

Can be downloaded to zookeeper official website

Http://zookeeper.apache.org/releases.html

3, Configuration zookeeper

(1) After the download is complete, come to the Conf folder, there is a zoo_sample.cfg official default configuration file. Copy one, rename to Zoo.cfg

(2) Configure, open zoo.cfg modify configuration information

#存储内存中数据库快照的位置, if you do not set a parameter, the update transaction log will be stored in the default location. DataDir=.. /zkdata# log file location Datalogdir=: /zklog# Listening Port ClientPort=2181

(3) Cluster configuration

server.1=127.0.0.1:12888:1388server. 2=127.0.0.1:12889:1389server. 3=127.0.0.1:12887:1387

Format: Server. A = B:c:d

A: is a number that indicates the first server

B: Server IP Address

C: is a port number that is used to exchange information for the cluster members, indicating that the server is exchanging information with the leader server in the cluster.

D: The port used exclusively for election leader when the leader is hung out

The complete configuration file is as follows

Copy two parts zookeeper unzip the configured folder, named

Under the corresponding file, modify the Zoo.cfg listening port address For example: The first zookeeper-3.4.6 program modifies the zoo.cfg configuration file

clientport=2181

The second Zookeeper-3.4.6-2 program modifies the zoo.cfg configuration file

clientport=2182

The third Zookeeper-3.4.6-2 program modifies the zoo.cfg configuration file

clientport=2183

Create ServerID

Create a new myID file under the configured DataDir directory, the file content is the corresponding ID number,

Like what:

zookeeper-3.4.6 program myID file has a content of 1

Zookeeper-3.4.6-2 program myID file has a content of 2

Zookeeper-3.4.6-3 program myID file has a content of 3

The directory I configured here is

Start Zookeeper

Start in the corresponding bin directory

Zkserver.cmd

third, installation Kafka(1) Download

To the official website http://kafka.apache.org/download can be downloaded here

Https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.2.2/kafka_2.9.2-0.8.2.2.tgz

This version

(2) configuration

After decompression to config folder to open the Server.properties configuration file to configure

(3) Configuration content

Modify or add the following configuration information

#唯一标识broker. id=0# Listening port port=9092host.name=127.0.0.1# message maximum size message.max.bytes=50485760# number of configuration replicas default.replication.factor=2# gets the maximum size of the replica.fetch.max.bytes=50485760# queue where messages persist in the location, can be multiple directories, separated by commas log.dirs=/tmp/ kafka-logs# the default number of partitions num.partitions=2# corresponds to the three IP and port addresses of the zookeeper that you just configured zookeeper.connect= 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183

　

(4) Cluster configuration

Copy the two extracted files, named as follows

Modify partial configuration information

Modify in the corresponding server.properties

# uniquely identifies

Broker.id=0

Broker.id=1

broker.id=2

# Listening Port

port=9092

port=9093

port=9094

start the corresponding Kafka

Go to the Bin/windows directory to start Kafka and specify the configuration file

Kafka-server-start.bat. /.. /config/server.properties

If you encounter an error in Kafka during startup :

Unrecognized VM option ' usecompressedoops ' error:clould not to create the Java vritual machine. ERROR:A Fatal exception has occurres. Program would exit.

Solution:

Locate Bin/windows/kafka-run-class.bat file,

Find 112 rows or so

IF ["%kafka_jvm_performance_opts%"] EQU [""] (  set KAFKA_JVM_PERFORMANCE_OPTS=-SERVER-XX:+USECOMPRESSEDOOPS-XX: +useparnewgc-xx:+useconcmarksweepgc-xx:+cmsclassunloadingenabled-xx:+cmsscavengebeforeremark-xx:+ Disableexplicitgc-djava.awt.headless=true)

Remove -xx:+usecompressedoops

Test Cluster

(1) Create a topic

Kafka-topics.bat--create--zookeeper 127.0.0.1:2181--replication-factor 1--partitions 1--topic test

(2) To see if the creation was successful

Kafka-topics.bat--list--zookeeper localhost:2181

(3) Send a message

Kafka-console-producer.bat--broker-list localhost:9092 --topic testthis is a message

(4) Receiving messages

Different clients can receive messages stating that the configuration was successful

Kafka Cluster Setup (in Windows environment)