Kafka details II. how to configure a Kafka Cluster

Kafka cluster configuration is relatively simple. For better understanding, the following three configurations are introduced here.

• Single Node: A broker Cluster
• Single Node: cluster of multiple Brokers
• Multi-node: Multi-broker Cluster

1. Single-node single-broker instance Configuration
1. first, start the zookeeper service Kafka. It provides the script for starting zookeeper (in the Kafka/bin/directory) and the zookeeper configuration file (in the Kafka/config/directory ), first go to the main directory of Kafka (which can be found through the whereis Kafka command): [[email protected] kafka-0.8] # bin/zookeeper-server-start.sh config/zookeeper. some important properties of the properties zookeeper configuration file: # data directory where the zookeeper snapshot is stored. datadir =/tmp/zookeeper
# The port listening for client request
Clientport = 2181 by default, the zookeeper server listens to port 2181. For more information, see the zookeeper official website. 2. start Kafka broker to run the startup Kafka service script provided by Kafka: [[email protected] kafka-0.8] # bin/kafka-server-start.sh config/server. important Properties in the properties broker configuration file: # broker ID. the ID of each broker must be unique. broker. id = 0
# Directory for storing logs
Log. dir =/tmp/kafka8-logs
# Zookeeper connection string
Zookeeper. connect = localhost: 21813. create a topic [[email protected] kafka-0.8] # bin/kafka-create-topic.sh -- zookeeper localhost: 2181 -- replica 1 -- partition 1 -- Topic kafkatopic4. start a producer process with the producer client provided by Kafka to send messages [[email protected] kafka-0.8] # bin/kafka-console-producer.sh -- broker-list localhost: 9092 -- Topic kafkatopic contains two parameters:

• Broker-list: defines the broker address for the producer to push messages, in the form of <IP Address: Port>
• Topic: the topic that the producer sends.

Then you can enter some messages, such:
5. start a consumer instance to consume messages [[email protected] kafka-0.8] # bin/kafka-console-consumer.sh -- zookeeper localhost: 2181 -- Topic kafkatopic -- from-beginning when you execute this command, you can see the producer production messages printed on the console:
Consumer-related property configurations are stored in consumer. in the properties file, important attributes include: # consumer group ID (a string that uniquely identifies a set of consumers # within the same consumer group) groupid = test-consumer-group
# Zookeeper connection string
Zookeeper. Connect = localhost: 2181
Ii. Run multi-broker instances on a Single Node
1. start zookeeper as above 2. to start a Kafka broker on one machine to start multiple broker instances, you only need to prepare multiple servers. properties file. For example, we need to start two brokers on one machine: first, we need to prepare two servers. properties configuration file

1. Server-1
   1. Brokerid = 1
   2. Port = 9092
   3. Log. dir =/temp/kafka8-logs/broker1
2. Server-2
   1. Brokerid = 2
   2. Port = 9093
   3. Log. dir =/temp/kafka8-logs/broker2

Then we start a broker [[[email protected] kafka-0.8] # env jmx_port = 9999 bin/kafka-server-start.sh config/server-1.properties
[[Email protected] kafka-0.8] # env jmx_port = 10000 bin/kafka-server-start.sh config/server-2.properties we can see that starting is to specify a different JMX port for each broker, JMX port is mainly used for monitoring and troubleshooting using tools such as jconsole. creating a topic now we're creating a broker with two partition partitions and two backups: [[email protected] kafka-0.8] # bin/kafka-create-topic.sh -- zookeeper localhost: 2181 -- replica 2 -- partition 2 -- Topic othertopic4. start producer to send messages. If we want to use one producer to send messages to multiple brokers, the only thing that needs to be changed is In the broker-list property, specify the broker to be connected: [[email protected] kafka-0.8] # bin/kafka-console-producer.sh -- broker-list localhost: 9092, localhost: 9093 -- Topic othertopic if we want to send different producers to different brokers, we only need to configure the broker-list attribute for each producer. 5. Start a consumer to consume messages as with the previous command [[email protected] kafka-0.8] # bin/kafka-console-consumer.sh -- zookeeper localhost: 2181 -- Topic othertopic -- from-beginning Iii. cluster mode (multi-node and multi-instance)The above two configuration methods are introduced, and it is easy to understand the cluster configuration. For example, we need to configure the cluster as shown in: zookeeper configuration file (zookeeper. properties): the configuration file (server. properties): Start two instances on one node according to the configuration method of multiple instances in a single node. The difference is that the zookeeper connection string needs to connect the zookeeper of all nodes # zookeeper connection string
Zookeeper. Connect = node1: 2181, node2: 2181