I. Introduction
Apache Kafka is an open-source message system project developed by the Apache Software Foundation and written by Scala. Kafka was initially developed by LinkedIn and open-source in early 2011. He graduated from Apache incubator in October 2012. The goal of this project is to provide a unified, high-throughput, and low-Wait platform for real-time data processing.
Ii. installation environment
Kafka Server IP:
10.0.0.25 10.0.0.26
Operating System:
CentOS 6.5 x86_64
Required software packages:
jdk-1.7.0_65-fcs.x86_64 java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
Hosts table records
cat /etc/hosts10.0.0.25 kafka110.0.0.26 kafka2
3. Configure Kafka nodes (kafka1 and kafka2)
Configure kafka1
1. Unzip the kafka_2.10-0.8.1.1.tgz to kafka1
tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/cd /usr/local/mv kafka_2.10-0.8.1.1/ kafka
2. Configure/usr/local/Kafka/config/server. properties on kafka1.
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/server.properties broker.id=1port=9000num.network.threads=2 num.io.threads=8socket.send.buffer.bytes=1048576socket.receive.buffer.bytes=1048576socket.request.max.bytes=104857600log.dirs=/usr/local/kafka/kafka-logsnum.partitions=8log.retention.hours=168log.segment.bytes=536870912log.retention.check.interval.ms=60000log.cleaner.enable=falsezookeeper.connect=kafka1:2181,kafka2:2181zookeeper.connection.timeout.ms=1000000
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeperclientPort=2181maxClientCnxns=0
Configure kafka2
1. Unzip the kafka_2.10-0.8.1.1.tgz to kafka1
tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/cd /usr/local/mv kafka_2.10-0.8.1.1/ kafka
2. Configure/usr/local/Kafka/config/server. properties on kafka2.
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/server.properties broker.id=2port=9000num.network.threads=2 num.io.threads=8socket.send.buffer.bytes=1048576socket.receive.buffer.bytes=1048576socket.request.max.bytes=104857600log.dirs=/usr/local/kafka/kafka-logsnum.partitions=8log.retention.hours=168log.segment.bytes=536870912log.retention.check.interval.ms=60000log.cleaner.enable=falsezookeeper.connect=kafka1:2181,kafka2:2181zookeeper.connection.timeout.ms=1000000
egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeperclientPort=2181maxClientCnxns=0
4. Compile the Kafka startup and shell close scripts
cat /etc/init.d/kafka #!/bin/bashsource /etc/profile;function Stop(){ps -ef |grep kafka|grep -v grep |awk ‘{print $2}‘|xargs kill -9 }function Start(){/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &> /dev/null &sleep 1/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &> /dev/null &}case $1 in stop) Stop ;; start) Start ;; *) echo "Usage: `basename $0` {stop|start}" ;;esac
Usage:
# Start Kafka:/etc/init. d/Kafka start # disable Kafka:/etc/init. d/Kafka stop
5. Enable Kafka to start automatically
echo ‘/etc/init.d/kafka start > /dev/null ‘ >> /etc/rc.local
This article is from the "past with the wind" blog and will not be reposted!
Installation and configuration of Apache Kafka distributed Message Queue