Kafka Cluster management, state saving is realized through zookeeper, so we should build zookeeper cluster first
Zookeeper Cluster setup
First, the SOFTWARE environment:
The zookeeper cluster requires more than half of the node to survive to be externally serviced, so the n
. Start the Zookeeper service
Since zookeeper is already available in the Kafka package, the script that launches the Kafka (in the Kafka_2.10-0.8.2.2/bin directory) and the Zookeeper configuration file (in KAFKA_2.10-0.8.2.2/ Config directory):
[Root@master kafka_2.10-0.8.2
Http://blog.csdn.net/fenglailea/article/details/52458737#t3directory (?) [-]
Installing zookeeper
1 Direct installation zookeeper no need to compile
2 source code compilation installation Zookeeper
Installing the PHP Zookeeper extension
Note The latest version of
decompression. 2. Start Zookeeper service
Because zookeeper is already in the Kafka's compressed package, it provides a script to start Kafka (under the Kafka_2.10-0.8.2.2/bin directory) and Zookeeper configuration file (in KAFKA_2.10-0.8.2.2/ Config directory):
[Root@master kafka_2.10-0.8.2.2]# bin/
From: http://doc.okbase.net/QING____/archive/19447.htmlAlso refer to:http://blog.csdn.net/21aspnet/article/details/19325373Http://blog.csdn.net/unix21/article/details/18990123Kafka as a distributed log collection or system monitoring service, it is necessary for us to use it in a suitable situation. The deployment of Kafka includes the Zookeeper environment/kafka
Original address: Http://www.jianshu.com/p/dc4770fc34b6zookeeper cluster constructionKafka is to manage the cluster through zookeeper.Although a simple version of the zookeeper is included in the Kafka package, there is a limited sense of functionality. In the production environment, it is recommended to download the official zookeeper software directly.
Down
To demonstrate the effect of the cluster, a virtual machine (window 7) is prepared, and a single IP multi-node zookeeper cluster is built in the virtual machine (the same is true for multiple IP nodes), and Kafka is installed in both native (Win 7) and virtual machines.Pre-preparation instructions:1. Three zookeeper servers, the local installation of one as Serve
Zookeeper uses zookeeper-3.4.7.tar.gz and kafka_2.10-0.9.0.0.tgz. First, install JDK (jdk-7u9-linux-i586.tar.gz) and SSH. The IP addresses are allocated to kafka1 (192.168.56.136), kafka2 (192.168.56.137), and kafka3 (192.168.56.138 ). The following describes how to install SSH and how to build and use zookeeper and Kafka
version, through the Yun install Clustershell installation, will be prompted no package, the source of the Yum in the long-term no update, so use to Epel-release
installation command:
sudo yum install epel-release
Then the Yum install Clustershell can be installed by Epel.
1.2.2: Configuring Cluster groups
Vim/etc/clustershell/groups
Add a group name: server IP or Host
kafka:192.168.17.129 192.168.17.130 192.168.17.131 II:
1, preparation work 1.1, machine preparationserver1:10.40.33.11server2:10.40.33.12server3:10.40.33.131.2, port occupancy situationzookeeper:2181,3888,4888kafka:90921.3. Software PreparationJDK1.7.0_51 (latest version of kafka-0.8.2.1 recommended to use 1.7 and later versions of JDK) zookeeper3.4.5 (and above) kafka_2.11-0.8.2.1 (latest version)2, installation 2.1, installation zookeeper1. Download zookeeper:http://mirror.bit.edu.cn/apache/
First, download zookeeper and Kafkafrom the official website(the locally used version is zookeeper-3.3.6, kafka_2.11-1.0.0):Second, configure zookeeper and Kafka and start, basic zkcli command and Kafka Create Delete topic Command. 2.1 Configuration
evaluation. Or Try:help.Scala> : Quitc:\users\zyx>1.3.4. Thriftc:\users\zyx>thrift-versionThrift version 0.11.01.3.5. Zookeeper1.3.5.1. ConfigurationIn the D:\Project\ServiceMiddleWare\zookeeper-3.4.10\conf directory, create a zoo.cfg file that reads as followsticktime=2000datadir=d:/project/servicemiddleware/zookeeper-3.4.10/data/dbDatalogdir=d:/project/servicemiddleware/
Environment Description:
Os:ubuntu 16.04
Zookeeper:zookeeper 3.4.9
kafka:kafka_2.11-0.11.0.0
JDK:JDK 8 (Kafka boot required to use to JDK)
Detailed instructions (all of the following actions are done under root user):One, JDK installationThe JDK is divided into the following categories: JRE, OPENJDK, Oracle JDK, where we install Oracle JDK (recommended installation)Add-apt-repository ppa:webupd8team/javaapt-get updateapt-get
The previous section was completed hive, this section we will build zookeeper, mainly behind the Kafka need to run on it.Zookeeper Download and installDownload Zookeeper 3.4.5 software package, can be downloaded in Baidu Network disk. Link: http://pan.baidu.com/s/1gePE9O3 Password: UNMT.Download finished with xftp upload to Spark1 server, I was placed in the/home
Zookeeper + kafka cluster installation 2
This is the continuation of the previous article. The installation of kafka depends on zookeeper. Both this article and the previous article are true distributed installation configurations and can be directly used in the production environment.
For
synchronization is complete.ClientPort: Listen for the service port of the client connection, if you have multiple zookeeper servers installed on one server, you need to set a different port number.DataDir: Memory Database snapshot address, transaction log address (unless otherwise specified by Datalogdir).2. Create a new file myID under $datadir and write to the server identification number#/tmp/zookeeper
Zookeeper + Kafka cluster Installation
Record the Zookeeper + Kafka cluster installation process in detail. Prepare three virtual machines. The system is RHEL64.
1) the configuration of each machine is as follows:
$ Cat/etc/hosts
...# Zookeeper hostnames:192.168.8.182 zk1192
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.