Kafka Environment build 2-broker cluster +zookeeper cluster (turn)

Original address: Http://www.jianshu.com/p/dc4770fc34b6zookeeper cluster construction

Kafka is to manage the cluster through zookeeper.
Although a simple version of the zookeeper is included in the Kafka package, there is a limited sense of functionality. In the production environment, it is recommended to download the official zookeeper software directly.

• Download the latest version of zookeeper software

  http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz

• Unzip and move the extracted files to/var/zookeeper and rename them to zookeeper00

  tar -xzf zookeeper-3.4.8.tar.gzmv zookeeper-3.4.8 /var/zookeeper/cd /var/zookeeper/mv zookeeper-3.4.8 zookeeper00

• Go to the/var/zookeeper/zookeeper00/conf directory and change zoo_sample.cfg to Zoo.cfg (because the default read profile is zoo.cfg when Zookeeper starts)

  cd /var/zookeeper/zookeeper00/confmv zoo_sample.cfg zoo.cfg

• Zookeeper Configuration file Modification

  Ticktime=2000#心跳时间, Unit: milliseconds initlimit=10#Follower在启动时需要在Synchronize data from leader within 10 heartbeat time synclimit=5 #超过  5 heartbeat time does not receive follower response, think this follower has been offline datadir=/ Zyxx_data/zookeeper/data00  #zookeeper存储数据的目录clientPort = 2181  #zookeeper服务端口server .0 =192.168.6.56: 20881:30881server.1=192.168.6.56: 20882: 30882server.2=192.168.6.56:20883:30883  

  • server.0, Server.1, server.2 refers to the list of nodes in the entire zookeeper cluster. The configuration rules for the server are: server. N=yyy:a:b
  • n indicates the server number
  • YYY represents the IP address of the server
  • A is the LF communication port, which represents the information exchanged between the server and the leader in the cluster.
  • B is the election port, which indicates the port to which the server communicates with each other when the new leader is elected (when leader hangs, the rest of the servers communicate with each other and select a new leader)
  • In general, the a port for each server in the cluster is the same, and the B port for each server is the same. However, when used as a pseudo-cluster, the IP address is the same, only a port and B port is not the same.

• Zookeeper Multi-node configuration

  Through the above configuration, a zookeeper node is ready, the following configuration of multiple nodes, we put zookeeper root directory zookeeper00 Copy multiple copies and rename

  -a zookeeper00/ zookeeper01cp -a zookeeper00/ zookeeper02

  For Zookeeper01/conf/zoo.cfg and Zookeeper02/conf/zoo.cfg, it is also necessary to modify the above several places. The modified parameters are as follows (because the cluster is emulated on the same machine, DataDir, ClientPort cannot be the same)
  Zookeeper01/conf/zoo.cfg

  ticktime=2000initlimit= 10synclimit=5 datadir=/zyxx_data/zookeeper/data01 clientPort =2182server.0=192.168.6.56:20881: 30881server.1=192.168.6.56:20882:30882server .2=192.168.6 .56:20883:30883    

  zookeeper02/conf/zoo.cfg

  ticktime=2000initlimit= 10synclimit=5 datadir=/zyxx_data/zookeeper/data02 clientPort =2183server.0=192.168.6.56:20881: 30881server.1=192.168.6.56:20882:30882server .2=192.168.6 .56:20883:30883    

• Zookeeper Data Catalog creation

  Zookeeper three nodes corresponding data directories are

  /zyxx_data/zookeeper/data00
  /zyxx_data/zookeeper/data01
  /zyxx_data/zookeeper/data02

  After creating the corresponding directory, in each of the three directories to create a file named myID, the file content only a number, representing the unique ID of the zookeeper node, that is, to ensure that the ID is unique within the cluster, and with the configuration file server.0, Server.1, The server.2 corresponds to.

Kafka Broker Cluster Setup

• Kafka Download, unzip

  wget http://apache.opencas.org/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgztar -xzf kafka_2.11-0.9.0.0.tgz

• Kafka Broker configuration file modification Modify config/server.properties under Kafka root directory

  Broker.id=0#整个集群内唯一id号, integers, generally from0 start Listeners=plaintext:192.168.6.56:9092 #协议, current broker machine IP, port, this value can be configured multiple, should be related to SSL, etc., more usage has not been understood, here is modified to IP and port. port=9092  #broker端口host. Name= 192.168.6.56  #broker machine ip< Span class= "hljs-built_in" >log.dirs=/zyxx_data/kafka-logs/kafka00 # Kafka the directory where the data is stored zookeeper.connect=192.168.6 .56:2181,192.168 .6.56:2182, 192.168.6.56: 2183  #zookeeper cluster list       

• Kafka Broker multi-node configuration

  Kafka Multi-node configuration, you can copy multiple copies of the software catalog, like zookeeper, and modify the respective configuration files. Here's another way: the same software catalog program, but start with a different configuration file

  Starting multiple broker nodes with different profiles is only suitable for a pseudo-cluster under a single machine, and there is no point in a real cluster on multiple machines.

• The Kafka root directory under the config/server.properties copy two copies, the name is modified to: Server-1.properties, server-2.properties. Each of them modifies their configuration as follows:

  Config/server-1.properties

  broker.id=1 listeners=PLAINTEXT://192.168.6.56:9093port=9093 host.name=192.168.6.56log.dirs=/zyxx_data/kafka-logs/kafka01 zookeeper.connect=192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183

  Config/server-2.properties

  broker.id=2 listeners=PLAINTEXT://192.168.6.56:9094port=9094 host.name=192.168.6.56log.dirs=/zyxx_data/kafka-logs/kafka02 zookeeper.connect=192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183

Cluster boot

• Zookeeper cluster boot

  Go to the/var/zookeeper/directory and execute

  startzookeeper01/bin/zkServer.sh startzookeeper02/bin/zkServer.sh start

  We can see if zookeeper has been started with the following command

  grep zoo.cfg

  
  View Zookeeper Process information
• Kafka Cluster Boot

  Go to the Kafka directory and Execute

  bin/kafka-server-start.sh -daemon config/server.propertiesbin/kafka-server-start.sh -daemon config/server-1.propertiesbin/kafka-server-start.sh -daemon config/server-2.properties

  The "-daemon" parameter represents the boot of Kafka server as a daemon.
  The official website and the internet mostly give the start command is not "-daemon" parameters, such as: "Bin/kafka-server-start.sh config/server.properties &", but after this way startup, if the user exits the SSH connection, The process is likely to end, not exactly why.

  We can see if Kafka server is started with the following command

  grep config/server

  
  Viewing Kafka server process information

Test

For simplicity, the producer and consumer tests are initiated by the command line.

• Create a Theme

  Go to the Kafka directory and create the "TEST5" topic topic: Partition 3, Backup to 3

  192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183 --replication-factor 3 --partitions 3 --topic test5

  --zookeeper : List of zookeeper clusters, separated by commas. You can specify only one or a few zookeeper node lists without specifying a list of nodes in the Zookeeper entire cluster .
  replication-factor : Number of copies, provide failover mechanism; 1 represents a data record on only one broker, and the general value is greater than 1, which means that one copy of the data is automatically synced to multiple brokers Prevent data loss after a broker goes down.
  partitions : A topic can be cut into multiple partitions, a consumer can consume multiple partitions, but a partitions can only be consumed by one consumer, So increasing partitions can increase the throughput of consumers. Kafka only guarantees that messages within a partitions are ordered, and that data between multiple partitions is unordered.

  View "TEST5" topic details

  bin/kafka-topics.sh --describe --zookeeper  192.168.6.56:2181 --topic test5

  
  TEST5 Topic Details
• start producer, consumer

  Start producer

  bin/kafka-console-producer.sh--broker-list 192.168.6 .56:9092--topic test5       

  --broker-list : value can be one or more nodes in the broker cluster

  Start the consumer (in the New SSH Connection window)

  bin/kafka-console-consumer.sh--zookeeper 192.168.6.56:
   2181--topic test5--from-beginning      

  --zookeeper : value can be one or more nodes in the Zookeeper cluster

  We can find that in the producer under the input of a message, enter after the consumer can output, indicating that we build the cluster no problem

  
  Producers and Consumers

• Simulating a node outage in a Kafka broker cluster

  We simulate it directly through the kill process.

  
  

  Re-observation of producers and consumers

  
  Producer consumers after a broker node is down

  We can see that consumers have a warning message, but they can still consume data.

• Simulating a node outage in a zookeeper cluster

  We simulate it directly through the kill process.

  
  

  Re-observation of producers and consumers

  
  The producer and consumer of a zookeeper node after the outage

  We can see that consumers can still consume data.
  This means that our entire cluster environment has been built successfully.

  Note that when you start a consumer, you specify only one zookeeper node in the command, and when you simulate a zookeeper node down, you kill the node exactly. In this case your consumer will keep the alarm information, but does not affect consumer information.

Wen/heichong (author of Jane's book)
Original link: http://www.jianshu.com/p/dc4770fc34b6
Copyright belongs to the author, please contact the author to obtain authorization, and Mark "book author".

Kafka Environment build 2-broker cluster +zookeeper cluster (turn)