Kafka Environment build 2-broker cluster +zookeeper cluster (turn)

Source: Internet
Author: User
Tags zookeeper software

Original address: Http://www.jianshu.com/p/dc4770fc34b6zookeeper cluster construction

Kafka is to manage the cluster through zookeeper.
Although a simple version of the zookeeper is included in the Kafka package, there is a limited sense of functionality. In the production environment, it is recommended to download the official zookeeper software directly.

  • Download the latest version of zookeeper software
    http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
  • Unzip and move the extracted files to/var/zookeeper and rename them to zookeeper00
    tar -xzf zookeeper-3.4.8.tar.gzmv zookeeper-3.4.8 /var/zookeeper/cd /var/zookeeper/mv zookeeper-3.4.8 zookeeper00
  • Go to the/var/zookeeper/zookeeper00/conf directory and change zoo_sample.cfg to Zoo.cfg (because the default read profile is zoo.cfg when Zookeeper starts)
    cd /var/zookeeper/zookeeper00/confmv zoo_sample.cfg zoo.cfg
  • Zookeeper Configuration file Modification
    Ticktime=2000#心跳时间, Unit: milliseconds initlimit=10#Follower在启动时需要在Synchronize data from leader within 10 heartbeat time synclimit=5 #超过  5 heartbeat time does not receive follower response, think this follower has been offline datadir=/ Zyxx_data/zookeeper/data00  #zookeeper存储数据的目录clientPort = 2181  #zookeeper服务端口server .0 =192.168.6.56: 20881:30881server.1=192.168.6.56: 20882: 30882server.2=192.168.6.56:20883:30883  
    • server.0, Server.1, server.2 refers to the list of nodes in the entire zookeeper cluster. The configuration rules for the server are: server. N=yyy:a:b
    • n indicates the server number
    • YYY represents the IP address of the server
    • A is the LF communication port, which represents the information exchanged between the server and the leader in the cluster.
    • B is the election port, which indicates the port to which the server communicates with each other when the new leader is elected (when leader hangs, the rest of the servers communicate with each other and select a new leader)
    • In general, the a port for each server in the cluster is the same, and the B port for each server is the same. However, when used as a pseudo-cluster, the IP address is the same, only a port and B port is not the same.
  • Zookeeper Multi-node configuration

    Through the above configuration, a zookeeper node is ready, the following configuration of multiple nodes, we put zookeeper root directory zookeeper00 Copy multiple copies and rename

    -a zookeeper00/ zookeeper01cp -a zookeeper00/ zookeeper02

    For Zookeeper01/conf/zoo.cfg and Zookeeper02/conf/zoo.cfg, it is also necessary to modify the above several places. The modified parameters are as follows (because the cluster is emulated on the same machine, DataDir, ClientPort cannot be the same)
    Zookeeper01/conf/zoo.cfg

    ticktime=2000initlimit= 10synclimit=5 datadir=/zyxx_data/zookeeper/data01 clientPort =2182server.0=192.168.6.56:20881: 30881server.1=192.168.6.56:20882:30882server .2=192.168.6 .56:20883:30883    

    zookeeper02/conf/zoo.cfg

    ticktime=2000initlimit= 10synclimit=5 datadir=/zyxx_data/zookeeper/data02 clientPort =2183server.0=192.168.6.56:20881: 30881server.1=192.168.6.56:20882:30882server .2=192.168.6 .56:20883:30883    
  • Zookeeper Data Catalog creation

    Zookeeper three nodes corresponding data directories are

    /zyxx_data/zookeeper/data00
    /zyxx_data/zookeeper/data01
    /zyxx_data/zookeeper/data02

    After creating the corresponding directory, in each of the three directories to create a file named myID, the file content only a number, representing the unique ID of the zookeeper node, that is, to ensure that the ID is unique within the cluster, and with the configuration file server.0, Server.1, The server.2 corresponds to.


Kafka Broker Cluster Setup
  • Kafka Download, unzip
    wget http://apache.opencas.org/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgztar -xzf kafka_2.11-0.9.0.0.tgz
  • Kafka Broker configuration file modification Modify config/server.properties under Kafka root directory
    Broker.id=0#整个集群内唯一id号, integers, generally from0 start Listeners=plaintext:192.168.6.56:9092 #协议, current broker machine IP, port, this value can be configured multiple, should be related to SSL, etc., more usage has not been understood, here is modified to IP and port. port=9092  #broker端口host. Name= 192.168.6.56  #broker machine ip< Span class= "hljs-built_in" >log.dirs=/zyxx_data/kafka-logs/kafka00 # Kafka the directory where the data is stored zookeeper.connect=192.168.6 .56:2181,192.168 .6.56:2182, 192.168.6.56: 2183  #zookeeper cluster list       
  • Kafka Broker multi-node configuration

    Kafka Multi-node configuration, you can copy multiple copies of the software catalog, like zookeeper, and modify the respective configuration files. Here's another way: the same software catalog program, but start with a different configuration file

    Starting multiple broker nodes with different profiles is only suitable for a pseudo-cluster under a single machine, and there is no point in a real cluster on multiple machines.

  • The Kafka root directory under the config/server.properties copy two copies, the name is modified to: Server-1.properties, server-2.properties. Each of them modifies their configuration as follows:

    Config/server-1.properties

    broker.id=1 listeners=PLAINTEXT://192.168.6.56:9093port=9093 host.name=192.168.6.56log.dirs=/zyxx_data/kafka-logs/kafka01 zookeeper.connect=192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183

    Config/server-2.properties

    broker.id=2 listeners=PLAINTEXT://192.168.6.56:9094port=9094 host.name=192.168.6.56log.dirs=/zyxx_data/kafka-logs/kafka02 zookeeper.connect=192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183
Cluster boot
  • Zookeeper cluster boot

    Go to the/var/zookeeper/directory and execute

    startzookeeper01/bin/zkServer.sh startzookeeper02/bin/zkServer.sh start

    We can see if zookeeper has been started with the following command

    grep zoo.cfg

    View Zookeeper Process information
  • Kafka Cluster Boot

    Go to the Kafka directory and Execute

    bin/kafka-server-start.sh -daemon config/server.propertiesbin/kafka-server-start.sh -daemon config/server-1.propertiesbin/kafka-server-start.sh -daemon config/server-2.properties

    The "-daemon" parameter represents the boot of Kafka server as a daemon.
    The official website and the internet mostly give the start command is not "-daemon" parameters, such as: "Bin/kafka-server-start.sh config/server.properties &", but after this way startup, if the user exits the SSH connection, The process is likely to end, not exactly why.

    We can see if Kafka server is started with the following command

    grep config/server

    Viewing Kafka server process information
Test

For simplicity, the producer and consumer tests are initiated by the command line.

  • Create a Theme

    Go to the Kafka directory and create the "TEST5" topic topic: Partition 3, Backup to 3

    192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183 --replication-factor 3 --partitions 3 --topic test5

    --zookeeper : List of zookeeper clusters, separated by commas. You can specify only one or a few zookeeper node lists without specifying a list of nodes in the Zookeeper entire cluster .
    replication-factor : Number of copies, provide failover mechanism; 1 represents a data record on only one broker, and the general value is greater than 1, which means that one copy of the data is automatically synced to multiple brokers Prevent data loss after a broker goes down.
    partitions : A topic can be cut into multiple partitions, a consumer can consume multiple partitions, but a partitions can only be consumed by one consumer, So increasing partitions can increase the throughput of consumers. Kafka only guarantees that messages within a partitions are ordered, and that data between multiple partitions is unordered.

    View "TEST5" topic details

    bin/kafka-topics.sh --describe --zookeeper  192.168.6.56:2181 --topic test5

    TEST5 Topic Details
  • start producer, consumer

    Start producer

    bin/kafka-console-producer.sh--broker-list 192.168.6 .56:9092--topic test5       

    --broker-list : value can be one or more nodes in the broker cluster

    Start the consumer (in the New SSH Connection window)

    bin/kafka-console-consumer.sh--zookeeper 192.168.6.56:
     2181--topic test5--from-beginning      

    --zookeeper : value can be one or more nodes in the Zookeeper cluster

    We can find that in the producer under the input of a message, enter after the consumer can output, indicating that we build the cluster no problem


    Producers and Consumers
    • Simulating a node outage in a Kafka broker cluster

      We simulate it directly through the kill process.


      Re-observation of producers and consumers


      Producer consumers after a broker node is down

      We can see that consumers have a warning message, but they can still consume data.

    • Simulating a node outage in a zookeeper cluster

      We simulate it directly through the kill process.


      Re-observation of producers and consumers


      The producer and consumer of a zookeeper node after the outage

      We can see that consumers can still consume data.
      This means that our entire cluster environment has been built successfully.

      Note that when you start a consumer, you specify only one zookeeper node in the command, and when you simulate a zookeeper node down, you kill the node exactly. In this case your consumer will keep the alarm information, but does not affect consumer information.



Wen/heichong (author of Jane's book)
Original link: http://www.jianshu.com/p/dc4770fc34b6
Copyright belongs to the author, please contact the author to obtain authorization, and Mark "book author".

Kafka Environment build 2-broker cluster +zookeeper cluster (turn)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.