Linux under Kafka Stand-alone installation configuration method (text) _linux

Introduced

Kafka is a distributed, partitioned, replicable messaging system. It provides the functionality of a common messaging system, but has its own unique design. What does this unique design look like?

Let's first look at a few basic messaging system terms:

Kafka the message to topic as a unit.
• The program that will release the message to Kafka topic becomes producers.
• The process of subscribing to topics and consuming messages becomes consumer.
Kafka is run as a cluster and can consist of one or more services, each of which is called a broker.
Producers sends messages to the Kafka cluster over the network, and the cluster provides messages to consumers, as shown in the following illustration:

The client and the server communicate through the TCP protocol. Kafka provides Java clients and provides support for multiple languages.

Description

Operating system: CentOS 6.x 64-bit

Kafka version: kafka_2.11-0.8.2.1

To achieve the purpose:

Stand-alone installation Configuration Kafka

Specific actions:

First, close SELinux, open firewall 9092 port

1. Close SELinux

Vi/etc/selinux/config

#SELINUX =enforcing #注释掉

#SELINUXTYPE =targeted #注释掉

selinux=disabled #增加

: wq! #保存退出

Setenforce 0 #使配置立即生效

2, configure the firewall, open 9092 ports

Vi/etc/sysconfig/iptables #编辑防火墙配置文件

# Firewall configuration written by System-config-firewall

# Manual Customization of this file is not recommended.

*filter

: INPUT ACCEPT [0:0]

: FORWARD ACCEPT [0:0]

: OUTPUT ACCEPT [0:0]

-A input-m state--state established,related-j ACCEPT

-A input-p icmp-j ACCEPT

-A input-i lo-j ACCEPT

-A input-m state--state new-m tcp-p TCP--dport 22-j ACCEPT

-A input-m state--state new-m tcp-p TCP --dport 9092-j ACCEPT

-A input-j REJECT--reject-with icmp-host-prohibited

-A forward-j REJECT--reject-with icmp-host-prohibited

COMMIT

: wq! #保存退出

Service iptables restart #最后重启防火墙使配置生效

Second, the installation of JDK

Kafka running requires JDK support

1. Download JDK

http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.rpm

Note: Copy directly to the download tool for download , version please use JDK7,JDK8 may not be compatible kafka_2.11-0.8.2.1

When the download is complete, upload it to the/USR/LOCAL/SRC directory

2. Install JDK

Cd/usr/local/src

chmod +x jdk-7u79-linux-x64.rpm # Add Execute Permissions

RPM-IVH jdk-7u79-linux-x64.rpm #安装

After the installation is complete, you can cd/usr/java/to the installation directory to view

3. Adding JDK to System environment variables

Vi/etc/profile #编辑, add the following code at the end

java_home=/usr/java/jdk1.7.0_79

Path= $PATH: $JAVA _home/bin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/x11r6/bin

Classpath=.: $JAVA _home/lib/tools.jar: $JAVA _home/lib/dt.jar

Export Java_home

Export PATH

Export CLASSPATH

: wq! #保存退出

Source/etc/profile #使配置文件立即生效

Java-version #查看JDK版本信息

To this end, the JDK installation is complete.

Iii. installation of Kafka

1, download Kafka

Cd/usr/local/src

wget http://archive.apache.org/dist/kafka/0.8.2.1/kafka_2.11-0.8.2.1.tgz

Note that the KAFKA_2.11-0.8.2.1.TGZ version is a compiled version that can be used for decompression.

TAR-XZVF kafka_2.11-0.8.2.1.tgz #解压

MV Kafka_2.11-0.8.2.1/usr/local/kafka #移动到安装目录

2. Configure Kafka

Mkdir/usr/local/kafka/log/kafka #创建kafka日志目录

Cd/usr/local/kafka/config #进入配置目录

VI server.properties #编辑修改相应的参数

Broker.id=0

port=9092 #端口号

host.name=192.168.0.11 #服务器IP地址, modify the IP for your own server

Log.dirs=/usr/local/kafka/log/kafka #日志存放路径, the directory created above

zookeeper.connect=localhost:2181 #zookeeper地址和端口, single configuration deployment, localhost:2181

: wq! #保存退出

3. Configure Zookeeper

Mkdir/usr/local/kafka/zookeeper #创建zookeeper目录

Mkdir/usr/local/kafka/log/zookeeper #创建zookeeper日志目录

Cd/usr/local/kafka/config #进入配置目录

VI zookeeper.properties #编辑修改相应的参数

Datadir=/usr/local/kafka/zookeeper #zookeeper数据目录

Datalogdir=/usr/local/kafka/log/zookeeper #zookeeper日志目录

clientport=2181

maxclientcnxns=100

ticktime=2000

initlimit=10

Synclimit=5

: wq! #保存退出

Create startup, shutdown Kafka scripts

Cd/usr/local/kafka

#创建启动脚本

VI kafkastart.sh #编辑, add the following code

#!/bin/sh

#启动zookeeper

/usr/local/kafka/bin/zookeeper-server-start.sh/usr/local/kafka/config/zookeeper.properties &

Sleep 3 #等3秒后执行

#启动kafka

/usr/local/kafka/bin/kafka-server-start.sh/usr/local/kafka/config/server.properties &

: wq! #保存退出

#创建关闭脚本

VI kafkastop.sh #编辑, add the following code

#!/bin/sh

#关闭zookeeper

/usr/local/kafka/bin/zookeeper-server-stop.sh/usr/local/kafka/config/zookeeper.properties &

Sleep 3 #等3秒后执行

#关闭kafka

/usr/local/kafka/bin/kafka-server-stop.sh/usr/local/kafka/config/server.properties &

: wq! #保存退出

#添加脚本执行权限

chmod +x kafkastart.sh

chmod +x kafkastop.sh

Five, set script to boot automatically execution

Vi/etc/rc.d/rc.local #编辑, adding a row at the end

Sh/usr/local/kafka/kafkastart.sh & # Set up to run the script in the background automatically

: wq! #保存退出

sh/usr/local/kafka/kafkastart.sh #启动kafka

sh/usr/local/kafka/kafkastop.sh #关闭kafka

At this point, Linux under the Kafka stand-alone installation configuration completed.

Extended reading:

Kafka Create topic

/usr/local/kafka/bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic Test

/usr/local/kafka/bin/kafka-topics.sh--list--zookeeper localhost:2181 test

/usr/local/kafka/bin/kafka-console-producer.sh--broker-list localhost:9092--topic test--from-beginning