Original address: Http://www.jianshu.com/p/dc4770fc34b6zookeeper cluster construction
Kafka is to manage the cluster through zookeeper.
Although a simple version of the zookeeper is included in the Kafka package, there is a limited sense of functionality. In the production environment, it is recommended to download the official zookeeper software directly.
- Download the latest version of zookeeper software
http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz
- Unzip and move the extracted files to/var/zookeeper and rename them to zookeeper00
tar -xzf zookeeper-3.4.8.tar.gzmv zookeeper-3.4.8 /var/zookeeper/cd /var/zookeeper/mv zookeeper-3.4.8 zookeeper00
- Go to the/var/zookeeper/zookeeper00/conf directory and change zoo_sample.cfg to Zoo.cfg (because the default read profile is zoo.cfg when Zookeeper starts)
cd /var/zookeeper/zookeeper00/confmv zoo_sample.cfg zoo.cfg
- Zookeeper Configuration file Modification
Ticktime=2000#心跳时间, Unit: milliseconds initlimit=10#Follower在启动时需要在Synchronize data from leader within 10 heartbeat time synclimit=5 #超过 5 heartbeat time does not receive follower response, think this follower has been offline datadir=/ Zyxx_data/zookeeper/data00 #zookeeper存储数据的目录clientPort = 2181 #zookeeper服务端口server .0 =192.168.6.56: 20881:30881server.1=192.168.6.56: 20882: 30882server.2=192.168.6.56:20883:30883
- server.0, Server.1, server.2 refers to the list of nodes in the entire zookeeper cluster. The configuration rules for the server are: server. N=yyy:a:b
- n indicates the server number
- YYY represents the IP address of the server
- A is the LF communication port, which represents the information exchanged between the server and the leader in the cluster.
- B is the election port, which indicates the port to which the server communicates with each other when the new leader is elected (when leader hangs, the rest of the servers communicate with each other and select a new leader)
- In general, the a port for each server in the cluster is the same, and the B port for each server is the same. However, when used as a pseudo-cluster, the IP address is the same, only a port and B port is not the same.
- Zookeeper Multi-node configuration
Through the above configuration, a zookeeper node is ready, the following configuration of multiple nodes, we put zookeeper root directory zookeeper00 Copy multiple copies and rename
-a zookeeper00/ zookeeper01cp -a zookeeper00/ zookeeper02
For Zookeeper01/conf/zoo.cfg and Zookeeper02/conf/zoo.cfg, it is also necessary to modify the above several places. The modified parameters are as follows (because the cluster is emulated on the same machine, DataDir, ClientPort cannot be the same)
Zookeeper01/conf/zoo.cfg
ticktime=2000initlimit= 10synclimit=5 datadir=/zyxx_data/zookeeper/data01 clientPort =2182server.0=192.168.6.56:20881: 30881server.1=192.168.6.56:20882:30882server .2=192.168.6 .56:20883:30883
zookeeper02/conf/zoo.cfg
ticktime=2000initlimit= 10synclimit=5 datadir=/zyxx_data/zookeeper/data02 clientPort =2183server.0=192.168.6.56:20881: 30881server.1=192.168.6.56:20882:30882server .2=192.168.6 .56:20883:30883
- Zookeeper Data Catalog creation
Zookeeper three nodes corresponding data directories are
/zyxx_data/zookeeper/data00
/zyxx_data/zookeeper/data01
/zyxx_data/zookeeper/data02
After creating the corresponding directory, in each of the three directories to create a file named myID, the file content only a number, representing the unique ID of the zookeeper node, that is, to ensure that the ID is unique within the cluster, and with the configuration file server.0, Server.1, The server.2 corresponds to.
Kafka Broker Cluster Setup
- Kafka Download, unzip
wget http://apache.opencas.org/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgztar -xzf kafka_2.11-0.9.0.0.tgz
- Kafka Broker configuration file modification Modify config/server.properties under Kafka root directory
Broker.id=0#整个集群内唯一id号, integers, generally from0 start Listeners=plaintext:192.168.6.56:9092 #协议, current broker machine IP, port, this value can be configured multiple, should be related to SSL, etc., more usage has not been understood, here is modified to IP and port. port=9092 #broker端口host. Name= 192.168.6.56 #broker machine ip< Span class= "hljs-built_in" >log.dirs=/zyxx_data/kafka-logs/kafka00 # Kafka the directory where the data is stored zookeeper.connect=192.168.6 .56:2181,192.168 .6.56:2182, 192.168.6.56: 2183 #zookeeper cluster list
- Kafka Broker multi-node configuration
Kafka Multi-node configuration, you can copy multiple copies of the software catalog, like zookeeper, and modify the respective configuration files. Here's another way: the same software catalog program, but start with a different configuration file
Starting multiple broker nodes with different profiles is only suitable for a pseudo-cluster under a single machine, and there is no point in a real cluster on multiple machines.
- The Kafka root directory under the config/server.properties copy two copies, the name is modified to: Server-1.properties, server-2.properties. Each of them modifies their configuration as follows:
Config/server-1.properties
broker.id=1 listeners=PLAINTEXT://192.168.6.56:9093port=9093 host.name=192.168.6.56log.dirs=/zyxx_data/kafka-logs/kafka01 zookeeper.connect=192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183
Config/server-2.properties
broker.id=2 listeners=PLAINTEXT://192.168.6.56:9094port=9094 host.name=192.168.6.56log.dirs=/zyxx_data/kafka-logs/kafka02 zookeeper.connect=192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183
Cluster boot
- Zookeeper cluster boot
Go to the/var/zookeeper/directory and execute
startzookeeper01/bin/zkServer.sh startzookeeper02/bin/zkServer.sh start
We can see if zookeeper has been started with the following command
grep zoo.cfg
View Zookeeper Process information
- Kafka Cluster Boot
Go to the Kafka directory and Execute
bin/kafka-server-start.sh -daemon config/server.propertiesbin/kafka-server-start.sh -daemon config/server-1.propertiesbin/kafka-server-start.sh -daemon config/server-2.properties
The "-daemon" parameter represents the boot of Kafka server as a daemon.
The official website and the internet mostly give the start command is not "-daemon" parameters, such as: "Bin/kafka-server-start.sh config/server.properties &", but after this way startup, if the user exits the SSH connection, The process is likely to end, not exactly why.
We can see if Kafka server is started with the following command
grep config/server
Viewing Kafka server process information
Test
For simplicity, the producer and consumer tests are initiated by the command line.
- Create a Theme
Go to the Kafka directory and create the "TEST5" topic topic: Partition 3, Backup to 3
192.168.6.56:2181,192.168.6.56:2182,192.168.6.56:2183 --replication-factor 3 --partitions 3 --topic test5
--zookeeper : List of zookeeper clusters, separated by commas. You can specify only one or a few zookeeper node lists without specifying a list of nodes in the Zookeeper entire cluster .
replication-factor : Number of copies, provide failover mechanism; 1 represents a data record on only one broker, and the general value is greater than 1, which means that one copy of the data is automatically synced to multiple brokers Prevent data loss after a broker goes down.
partitions : A topic can be cut into multiple partitions, a consumer can consume multiple partitions, but a partitions can only be consumed by one consumer, So increasing partitions can increase the throughput of consumers. Kafka only guarantees that messages within a partitions are ordered, and that data between multiple partitions is unordered.
View "TEST5" topic details
bin/kafka-topics.sh --describe --zookeeper 192.168.6.56:2181 --topic test5
TEST5 Topic Details
- start producer, consumer
Start producer
bin/kafka-console-producer.sh--broker-list 192.168.6 .56:9092--topic test5
--broker-list : value can be one or more nodes in the broker cluster
Start the consumer (in the New SSH Connection window)
bin/kafka-console-consumer.sh--zookeeper 192.168.6.56:
2181--topic test5--from-beginning
--zookeeper : value can be one or more nodes in the Zookeeper cluster
We can find that in the producer under the input of a message, enter after the consumer can output, indicating that we build the cluster no problem
Producers and Consumers
- Simulating a node outage in a Kafka broker cluster
We simulate it directly through the kill process.
Re-observation of producers and consumers
Producer consumers after a broker node is downWe can see that consumers have a warning message, but they can still consume data.
- Simulating a node outage in a zookeeper cluster
We simulate it directly through the kill process.
Re-observation of producers and consumers
The producer and consumer of a zookeeper node after the outageWe can see that consumers can still consume data.
This means that our entire cluster environment has been built successfully.
Note that when you start a consumer, you specify only one zookeeper node in the command, and when you simulate a zookeeper node down, you kill the node exactly. In this case your consumer will keep the alarm information, but does not affect consumer information.
Wen/heichong (author of Jane's book)
Original link: http://www.jianshu.com/p/dc4770fc34b6
Copyright belongs to the author, please contact the author to obtain authorization, and Mark "book author".
Kafka Environment build 2-broker cluster +zookeeper cluster (turn)