Kafka + Zookeeper cluster construction, kafkazookeeper Cluster

Kafka + Zookeeper cluster construction, kafkazookeeper Cluster

The last time I introduced how to build an elasticsearch cluster, I hope to help you. Here I will introduce the kafka cluster and the last build.

First, let's take a look at kafka and zookeeper?

Apache kafka is a distributed Message System Based on push-subscribe. It features fast, scalable, and persistent. It is now an open-source system under Apache. As part of the hadoop ecosystem, it is widely used by various commercial companies. Its biggest feature is its ability to process a large amount of data in real time to meet various needs scenarios: hadoop-based batch processing systems, low-latency real-time systems, and storm/spark stream Processing engines.

Features:

• High throughput and low latency: Kafka can process hundreds of thousands of messages per second, with a latency of only a few milliseconds.
• Scalability: Kafka clusters support hot expansion
• Durability and Reliability: The message is persisted to the local disk and supports data backup to prevent data loss.
• Fault Tolerance: Nodes in the cluster are allowed to fail (if the number of replicas is n, n-1 nodes are allowed to fail)
• High concurrency: Supports simultaneous read/write of thousands of clients

ZooKeeper is a distributed, open-source distributed application Coordination Service. It contains a simple primitive set. distributed applications can implement synchronization services, configuration maintenance, and naming services based on it.


Cluster role:

• The Leader server is the core of the zookeeper cluster's working mechanism.
• The Follower Server is a Follower of the zookeeper cluster status.
• The Observer server acts as an Observer.

Next, let's go directly to the question: how to correctly build kafka and zookeeper clusters.

1. zookeeper cluster configuration

1. Modify the Host Name

kafka1.example.com --> 172.16.81.131kafka2.example.com --> 172.16.81.132

2. Modify the hosts file

[root@kafka1 opt]# cat /etc/hosts　　127.0.0.1   kafka1.example.com localhost localhost.localdomain localhost4 localhost4.localdomain4　　::1         localhost localhost.localdomain localhost6 localhost6.localdomain6[root@kafka2 opt]# cat /etc/hosts　　127.0.0.1   kafka2.example.com localhost localhost.localdomain localhost4 localhost4.localdomain4　　::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

3. Install jdk

cd /optjdk-8u131-linux-x64.tar.gztar -zxvf jdk-8u131-linux-x64.tar.gzmv jdk-1.8.0_131 /usr/local/

4. Configure jdk Environment Variables

[Root @ kafka1 opt] # tail-10/etc/profile # JAVA environment variable export JAVA_HOME =/usr/local/jdk1.8.0 _ 131 export JAVA_BIN = $ JAVA_HOME/bin export JAVA_LIB = $ JAVA_HOME/lib export CLASSPATH =.: $ JAVA_LIB/tools. jar: $ JAVA_LIB/dt. jar export PATH = $ JAVA_BIN: $ PATH export _ JAVA_SR_SIGNUM = 12 # zookeeper environment variable export ZOOKEEPER_HOME =/opt/zookeeper/export PATH = $ ZOOKEEPER_HOME/bin: $ PATH export PATH [root @ kafka2 opt] # tail-10/etc/profile # JAVA environment variable export JAVA_HOME =/usr/local/jdk1.8.0 _ 131 export JAVA_BIN = $ JAVA_HOME/bin export JAVA_LIB = $ JAVA_HOME/lib export CLASSPATH =.: $ JAVA_LIB/tools. jar: $ JAVA_LIB/dt. jar export PATH = $ JAVA_BIN: $ PATH export _ JAVA_SR_SIGNUM = 12 # zookeeper environment variable export ZOOKEEPER_HOME =/opt/zookeeper/export PATH = $ ZOOKEEPER_HOME/bin: $ PATH export PATH # application environment variable source/etc/profile

5. download the software package

zookeeper-3.4.10.tar.gz

# Decompress
Tar-zxvf zookeeper-3.4.10.tar.gz
Music zookeeper-3.4.10 zookeeper
Cd/opt/zookeeper/config/
Cp zoo_sample.cfg zoo. cfg

6. Edit the zookeeper configuration file

[root@kafka1 opt]# cat /opt/zookeeper/conf/zoo.cfg | grep -v '^#' | grep -v '^$' tickTime=2000 initLimit=20 syncLimit=10 dataDir=/opt/data/zookeeper/data datalogDir=/opt/data/zookeeper/logs clientPort=2181 server.1=172.16.81.131:2888:3888 server.2=172.16.81.132:2888:3888[root@kafka2 opt]# cat /opt/zookeeper/conf/zoo.cfg | grep -v '^#' | grep -v '^$' tickTime=2000 initLimit=20 syncLimit=10 dataDir=/opt/data/zookeeper/data datalogDir=/opt/data/zookeeper/logs clientPort=2181 server.1=172.16.81.131:2888:3888 server.2=172.16.81.132:2888:3888

# Note: The zookeeper configuration file cannot be followed by comments. Otherwise, an error is reported!
# Note:
TickTime: This time is used as the interval between the Zookeeper server or between the client and the server to maintain the heartbeat, that is, each tickTime will send a heartbeat.
Port 2888: indicates the port on which the server exchanges information with the Leader server in the cluster;
Port 3888: In case the Leader server in the cluster fails, a port is required to re-elect and select a new Leader, this port is used to execute the election when the Server communicates with each other.

7. Create the datadir directory on the kafka1 and kafka2 servers respectively.

mkdir -p /opt/kafka/datamkdir -p /opt/kafka/data/zookeeper

8. Write IDs respectively

[Root @ kafka1 opt] # echo "1">/opt/kafka/data/zookeeper/myid [root @ kafka2 ~] # Echo "2">/opt/kafka/data/zookeeper/myid # note that the ID cannot be the same

9. Start the zookeeper Cluster

cd /opt/zookeeper/bin/zkServer.sh start

10. startup Effect

[Rootkafka1 ~] # Netstat-nlpt | grep-E "2181 | 2888 | 3888" tcp 0 0 ::: 2181 ::: * LISTEN 33644/javatcp 0 0 :: ffff: 10.1.1.247: 3888:: * LISTEN 33644/java [root @ kafka2 ~] # Netstat-nlpt | grep-E "2181 | 2888 | 3888" tcp 0 0 ::: 2181 ::: * LISTEN 35016/javatcp 0 0 :: ffff: 10.1.1.248: 2888:: * LISTEN 35016/java # which is the leader, then it has port 2888 tcp 0 0: ffff: 10.1.1.248: 3888 ::: * LISTEN 35016/java

Ii. Build a kafka Cluster

1. Configuration File

[Root @ kafka1 opt] # cat/opt/kafka/config/server. properties | grep-v '^ #' | grep-v '^ $' broker. id = 1 listeners = PLAINTEXT: // 172.16.81.131: 9092num. network. threads = 3num. io. threads = 8socket. send. buffer. bytes = 102400socket. receive. buffer. bytes = 102400socket. request. max. bytes = 104857600log. dirs =/opt/kafka/data/kafka-logsnum.partitions = 10num. recovery. threads. per. data. dir = 1offsets. topic. replication. factor = 1 transac Tion. state. log. replication. factor = 1transaction. state. log. min. isr = 1log. retention. hours = 168log. segment. bytes = bytes 3741824log. retention. check. interval. ms = 300000zookeeper. connect = 172.16.81.131: 2181,172.16 .81.132: 2181zookeeper. connection. timeout. ms = 6000group. initial. rebalance. delay. ms = 0 [root @ kafka2 ~] # Cat/opt/kafka/config/server. properties | grep-v '^ #' | grep-v '^ $' broker. id = 2 listeners = PLAINTEXT: // 172.16.81.132: 9092num. network. threads = 3num. io. threads = 8socket. send. buffer. bytes = 102400socket. receive. buffer. bytes = 102400socket. request. max. bytes = 104857600log. dirs =/opt/kafka/data/kafka-logsnum.partitions = 10num. recovery. threads. per. data. dir = 1offsets. topic. replication. factor = 1transaction. state. log. replication. factor = 1transaction. state. log. min. isr = 1log. retention. hours = 168log. segment. bytes = bytes 3741824log. retention. check. interval. ms = 300000zookeeper. connect = 172.16.81.131: 2181,172.16 .81.132: 2181zookeeper. connection. timeout. ms = 6000group. initial. rebalance. delay. ms = 0 # Note: broker. the id cannot be the same

2. Start the kafka Cluster

/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &

3. startup Effect

[root@kafka1 opt]# netstat -lntpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name          tcp6       0      0 :::47457                :::*                    LISTEN      6582/java           tcp6       0      0 172.16.81.131:9092      :::*                    LISTEN      9260/java           tcp6       0      0 :::2181                 :::*                    LISTEN      6582/java           tcp6       0      0 :::33230                :::*                    LISTEN      9260/java           tcp6       0      0 172.16.81.131:3888      :::*                    LISTEN      6582/java           [root@kafka2 ~]# netstat -lntpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name          tcp6       0      0 172.16.81.132:9092      :::*                    LISTEN      9395/java           tcp6       0      0 :::42884                :::*                    LISTEN      6779/java           tcp6       0      0 :::2181                 :::*                    LISTEN      6779/java           tcp6       0      0 172.16.81.132:2888      :::*                    LISTEN      6779/java           tcp6       0      0 172.16.81.132:3888      :::*                    LISTEN      6779/java                 tcp6       0      0 :::38557                :::*                    LISTEN      9395/java

4. Test whether zookeeper and kafka are normal.

(1) create a topic [root @ kafka2 ~] #/Opt/kafka/bin/kafka-topics.sh -- create -- zookeeper localhost: 2181 -- replication-factor 2 -- partitions 1 -- topic summerCreated topic "summer ". # Note: The factor size cannot exceed the number of brokers. Otherwise, an error is reported. The current cluster broker value is 2 (2). Check which topics have been created. [root @ kafka2 ~] #/Opt/kafka/bin/kafka-topics.sh -- list -- zookeeper 172.16.81.132: 2181 summer [root @ kafka1 ~] #/Opt/kafka/bin/kafka-topics.sh -- list -- zookeeper 172.16.81.131: 2181 summer (3) view topic details [root @ kafka2 ~] #/Opt/kafka/bin/kafka-topics.sh -- describe -- zookeeper localhost: 2181 -- topic summerTopic: summerPartitionCount: 1 ReplicationFactor: 2 Configs: Topic: summerPartition: 0 Leader: 2 Replicas: 2, 1Isr: # topic name: summer # Partition: Only one, starting from 0 # broker with leader: id 2 # Replicas copy exists on broker with id # Isr: the active broker (4) sends messages. The producer role [root @ kafka2 ~] is used here. #/Bin/bash/opt/kafka/bin/kafka-console-producer.sh -- broker-list localhost: 9092 -- topic summer> Hello, wangyanlin> I am from china. >>>;> ^ C [root @ kafka2 ~] # (5) receive messages. Here the consumer role [root @ kafka2 ~] is used. #/Opt/kafka/bin/kafka-console-consumer.sh -- zookeeper localhost: 2181 -- topic summer -- from-beginningUsing the role leconsumer with old consumer is deprecated and will be removed in a future major release. consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]. hello, wangyanlinI am from china .; ^ CProcessed a total of 5 messages [root @ kafka1 kafka] #/opt/kafka/bin/kafka-co Nsole-consumer.sh -- zookeeper 172.16.81.132: 2181 -- topic summer -- from-beginningUsing the role leconsumer with old consumer is deprecated and will be removed in a future major release. consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]. hello, wangyanlinI am from china .; ^ CProcessed a total of 5 messages (6) delete consumption topic/opt/kafka/bin/kafka-topics.sh -- delete -- zookeeper loc Alhost: 2181 -- enable delete. topic. enable in conf to true # The test is normal !! Done!

Test that the kafka cluster can receive consumption information and consumption information normally !!

Logstash log collection and filtering and kibana graphical display will be released in the future.