Kafka cluster Installation

Install a Kafka Cluster

Assume that four brokers need to be configured in the cluster to form the Kafka cluster in the following chart.

2.1 configuration file

Configure all Kafka producer files. The brokerpid value is a unique numeric value. The core attributes are as follows:

<Span style = "font-size: 18px;"> # the ID of the broker. this must be set to a unique integer for each broker. broker. id = 11 # The port the socketserver listens onport = 9092 # hostname the broker willbind. if not set, the server will bind to all interfaceshost. name = hadoop-master # zookeeper cluster zookeeper. connect = machine-1: 2222, machine-2: 2222, machine-0: 2222 </span>

Because Kafka depends on the zookeeper cluster, you must first start the zookeeper cluster. I will not introduce it here.

2.2 broker attribute Configuration

On machine-0 and hadoop-master, configure the corresponding broker configuration file. The configuration attributes of the two machines are the same.

On hadoop-master, the server. properties configuration of Kafka is as follows:

<span style="font-size:18px;"># The id of the broker.This must be set to a unique integer for each broker.broker.id=11 #############################Socket Server Settings ############################# # The port the socketserver listens onport=9092 # Hostname the broker willbind to. If not set, the server will bind to all interfaceshost.name=hadoop-master # Hostname the broker willadvertise to producers and consumers. If not set, it uses the# value for"host.name" if configured. Otherwise, it will use the value returned from#java.net.InetAddress.getCanonicalHostName().#advertised.host.name=





Server-1.properties configuration for Kafka on hadoop-MASTER:

<span style="font-size:18px;"># The id of the broker.This must be set to a unique integer for each broker.broker.id=12 #############################Socket Server Settings ############################# # The port the socketserver listens onport=9093 # Hostname the broker willbind to. If not set, the server will bind to all interfaceshost.name=hadoop-master # Hostname the broker willadvertise to producers and consumers. If not set, it uses the# value for"host.name" if configured. Otherwise, it will use the value returned from#java.net.InetAddress.getCanonicalHostName().#advertised.host.name=


2.3 Startup Script

Start the corresponding script on the two machines:

<span style="font-size:18px;">bin/kafka-server-start.shconfig/server.propertiesbin/kafka-server-start.shconfig/server-1.properties</span>





Create a topic:

<Span style = "font-size: 18px;" >## create topic cluster_topicbin/kafka-topics.sh -- create -- zookeepermachine-1: 2222, machine-0: 2222, machine-2: 2222 -- replication-Factor 3 -- partitions 3 -- topiccluster_topic </span>





The following is the shell script for producer and consumer. The interaction between the sender and receiver shows the subscription and publishing process. Start a producer to publish messages.

<span style="font-size:18px;">bin/kafka-console-producer.sh--broker-listhadoop-master:9092,hadoop-master:9093,machine-0:9092,machine-0:9093--topic cluster_topic</span>





The message to be sent will be sent to the specified four brokers. The following starts the consumer to accept the message.

<span style="font-size:18px;"> bin/kafka-console-consumer.sh --zookeepermachine-1:2222,machine-0:2222,machine-2:2222--topic cluster_topic --from-beginning</span>





The above script accepts messages from cluster_topic, which means that all messages sent to cluster_topic will be accepted by this consumer. Send the message displayed in the following chart, and the consumer will also print the corresponding message.

Assume that the machine-0 is dropped. When using this producer to send a message, you will find that the message will still be accepted. This also shows that the cluster implements the fault tolerance function.
The following describes several useful scripts to help us monitor Kafka information.
List topics

<span style="font-size:18px;">bin/kafka-topics.sh--list --zookeepermachine-1:2222,machine-0:2222,machine-2:2222</span>





Queries the execution thread of a configuration file, for example, the following queries the process information that the execution configuration file is server-1.properties.

<span style="font-size:18px;">ps | grep server-1.properties</span>


<span style="font-size:18px;">root     21765 21436  0 09:48 pts/0    00:00:00 grep server.propertiesroot     23156     1  0 Aug27 ?        00:06:11 /usr/java/latest/bin/java -Xmx1G -Xms1G -server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/root/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/root/kafka/bin/../logs -Dlog4j.configuration=file:/root/kafka/bin/../config/log4j.properties -cp :/root/kafka/bin/../core/build/dependant-libs-2.8.0/*.jar:/root/kafka/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/root/kafka/bin/../clients/build/libs//kafka-clients*.jar:/root/kafka/bin/../examples/build/libs//kafka-examples*.jar:/root/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/root/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/root/kafka/bin/../libs/jopt-simple-3.2.jar:/root/kafka/bin/../libs/kafka_2.10-0.8.1.1.jar:/root/kafka/bin/../libs/kafka_2.10-0.8.1.1-javadoc.jar:/root/kafka/bin/../libs/kafka_2.10-0.8.1.1-scaladoc.jar:/root/kafka/bin/../libs/kafka_2.10-0.8.1.1-sources.jar:/root/kafka/bin/../libs/log4j-1.2.15.jar:/root/kafka/bin/../libs/metrics-core-2.2.0.jar:/root/kafka/bin/../libs/scala-library-2.10.1.jar:/root/kafka/bin/../libs/slf4j-api-1.7.2.jar:/root/kafka/bin/../libs/snappy-java-1.0.5.jar:/root/kafka/bin/../libs/zkclient-0.3.jar:/root/kafka/bin/../libs/zookeeper-3.3.4.jar:/root/kafka/bin/../core/build/libs/kafka_2.8.0*.jar kafka.Kafka /root/kafka/bin/../config/server.properties</span>













Kafka cluster Installation