Installation and configuration of Apache Kafka distributed Message Queue

I. Introduction

Apache Kafka is an open-source message system project developed by the Apache Software Foundation and written by Scala. Kafka was initially developed by LinkedIn and open-source in early 2011. He graduated from Apache incubator in October 2012. The goal of this project is to provide a unified, high-throughput, and low-Wait platform for real-time data processing.

Ii. installation environment
Kafka Server IP:

10.0.0.25 10.0.0.26

Operating System:

CentOS 6.5 x86_64

Required software packages:

jdk-1.7.0_65-fcs.x86_64 java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

Hosts table records

cat /etc/hosts10.0.0.25 kafka110.0.0.26 kafka2

3. Configure Kafka nodes (kafka1 and kafka2)

Configure kafka1

1. Unzip the kafka_2.10-0.8.1.1.tgz to kafka1

tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/cd /usr/local/mv kafka_2.10-0.8.1.1/ kafka

2. Configure/usr/local/Kafka/config/server. properties on kafka1.

egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/server.properties broker.id=1port=9000num.network.threads=2 num.io.threads=8socket.send.buffer.bytes=1048576socket.receive.buffer.bytes=1048576socket.request.max.bytes=104857600log.dirs=/usr/local/kafka/kafka-logsnum.partitions=8log.retention.hours=168log.segment.bytes=536870912log.retention.check.interval.ms=60000log.cleaner.enable=falsezookeeper.connect=kafka1:2181,kafka2:2181zookeeper.connection.timeout.ms=1000000

egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeperclientPort=2181maxClientCnxns=0

Configure kafka2

1. Unzip the kafka_2.10-0.8.1.1.tgz to kafka1

tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/cd /usr/local/mv kafka_2.10-0.8.1.1/ kafka

2. Configure/usr/local/Kafka/config/server. properties on kafka2.

egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/server.properties broker.id=2port=9000num.network.threads=2 num.io.threads=8socket.send.buffer.bytes=1048576socket.receive.buffer.bytes=1048576socket.request.max.bytes=104857600log.dirs=/usr/local/kafka/kafka-logsnum.partitions=8log.retention.hours=168log.segment.bytes=536870912log.retention.check.interval.ms=60000log.cleaner.enable=falsezookeeper.connect=kafka1:2181,kafka2:2181zookeeper.connection.timeout.ms=1000000

egrep -v ‘(^#|^$)‘ /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeperclientPort=2181maxClientCnxns=0

4. Compile the Kafka startup and shell close scripts

cat /etc/init.d/kafka #!/bin/bashsource /etc/profile;function Stop(){ps -ef |grep kafka|grep -v grep |awk ‘{print $2}‘|xargs kill -9 }function Start(){/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &> /dev/null &sleep 1/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &> /dev/null &}case $1 in         stop)            Stop            ;;        start)            Start            ;;        *)            echo "Usage: `basename $0` {stop|start}"            ;;esac

Usage:

# Start Kafka:/etc/init. d/Kafka start # disable Kafka:/etc/init. d/Kafka stop

5. Enable Kafka to start automatically

echo ‘/etc/init.d/kafka start > /dev/null ‘ >> /etc/rc.local

This article is from the "past with the wind" blog and will not be reposted!

Installation and configuration of Apache Kafka distributed Message Queue