Kafka deployment and code instance

Kafka deployment and code instance

As a distributed log collection or System Monitoring Service, kafka must be used in a suitable scenario. The deployment of kafka includes the zookeeper environment and kafka environment, and some configuration operations are required. Next, we will introduce how to use kafka.

We use three zookeeper instances to build a zk cluster and two kafka brokers to build a kafka cluster.

Kafka is 0.8 V, and zookeeper is V.

-------------------------------------- Split line --------------------------------------

Kafka architecture design of the distributed publish/subscribe message system

Apache Kafka code example

Apache Kafka tutorial notes

Getting started with Kafka

-------------------------------------- Split line --------------------------------------

Test-kafka.zip (5.3 KB) download

Free in http://linux.bkjia.com/

The username and password are both www.bkjia.com

The specific download directory is available at/July 15,/July 29/July 29/Kafka deployment and code instance.

For the download method, see

-------------------------------------- Split line --------------------------------------

1. Build a Zookeeper Cluster

We have 3 zk instances, respectively for zk-0, zk-1, zk-2; if you are just testing to use, you can use 1 zk instance.

1) zk-0

Adjust the configuration file:

ClientPort = 2181
Server.0 = MAID: 2888: 3888
Server .1 = 127.0.0.1: 2889: 3889
Server.2 = 127.0.0.1: 2890: 3890
# You only need to modify the above configurations. Retain the default values for other configurations.

Start zookeeper

./ZkServer. sh start

2) zk-1

Tune the configuration file (one other configuration and one zk-0 ):

ClientPort = 2182
# You only need to modify the above configurations. Retain the default values for other configurations.

Start zookeeper

./ZkServer. sh start

3) zk-2

Tune the configuration file (one other configuration and one zk-0 ):

ClientPort = 2183
# You only need to modify the above configurations. Retain the default values for other configurations.

Start zookeeper

./ZkServer. sh start

Ii. Build a Kafka Cluster

Because the Broker configuration file involves the related conventions of zookeeper, we first display the broker configuration file. We use two kafka brokers to build this cluster environment, namely kafka-0, kafka-1.

1) kafka-0

Modify the configuration file in the config directory as follows:

Broker. id = 0
Port = 9092
Num. network. threads = 2
Num. io. threads = 2
Socket. send. buffer. bytes = 1048576
Socket. receive. buffer. bytes = 1048576
Socket. request. max. bytes = 104857600
Log. dir =./logs
Num. partitions = 2
Log. flush. interval. messages = 10000
Log. flush. interval. ms = 1000
Log. retention. hours = 168
# Log. retention. bytes = 1073741824
Log. segment. bytes = 536870912
# Replication mechanism, so that the partitions of each topic are backed up in the kafka-cluster
# Used to improve cluster fault tolerance capability ..
Default. replication. factor = 1
Log. cleanup. interval. mins = 10
Zookeeper. connect = 127.0.0.1: 2181,127.0 .0.1: 2182,127.0 .0.1: 2183
Zookeeper. connection. timeout. ms = 1000000

Because kafka is written in scala, you must first prepare scala-related environments to run kafka.

> Cd kafka-0
>./Sbt update
>./Sbt package
>./Sbt assembly-package-dependency

The execution of the last command may be abnormal for the moment. Start kafka broker:

> JMS_PORT = 9997 bin/kafka-server-start.sh config/server. properties &

Because the zookeeper environment is running normally, you do not need to mount and start zookeeper through kafka. If multiple kafka brokers are deployed on one machine, you need to declare JMS_PORT.

2) kafka-1

Broker. id = 1
Port = 9093
# Other configurations are consistent with the kafka-0

Then run the packaging command like the kafka-0 and then start the broker.

> JMS_PORT = 9998 bin/kafka-server-start.sh config/server. properties &

You can still run the following command to view the distribution and survival status of the topic "partition"/"replicas.

> Bin/kafka-list-topic.sh -- zookeeper localhost: 2181
Topic: my-replicated-topic partition: 0 leader: 2 replicas: 1, 2, 0 isr: 2
Topic: test partition: 0 leader: 0 replicas: 0 isr: 0 & nbsp;

So far the environment has been okay, so let's start to show the programming instance. [Configuration parameters]

For more details, please continue to read the highlights on the next page:

1 2 3 Next Page