The first experience of Kafka learning

Learning questions:

Does 1.kafka need zookeeper?
What is 2.kafka?
What concepts does 3.kafka contain?
4. How do I simulate a client sending and receiving a message preliminary test? (Kafka installation steps)
5.kafka cluster How to interact with zookeeper?

1.kafka Do I need zoopkeeper

Kafka applications require zookeeper, can use the zookeeper provided by the Kafka installation package, or they can be downloaded separately zookeeper

What 2.kafka is.

Kafka is a distributed messaging system. Kafka is a distributed, partitioned, replicable messaging system

3.kafka Common Concepts:

Product: Producer, the program that sends messages to Kafka, called the generator

Consumer: Consumer, a program that subscribes to messages from Kafka, is called a consumer

Topic:kafka is a group of topic to summarize the message, or a collection of good news

Broker:kafka's services. How Kafka works is a cluster run, and a cluster can consist of one or more services, each of which is called a broker

Partition: Partition. A top consists of one or more partitions. Partitions can move one TOPITC to multiple places for storage to improve parallel processing power.

Replication: Copy. A partition consists of one copy or multiple replicas. Replicas are used for partitioned backups.

4. Installation Steps
(1) Download the kafka_2.10-0.9.0.0.tgz package and put it in the/usr/local directory
Tar zxvf kafka_2.10-0.9.0.0.tgz
LN-SV kafka_2.10-0.9.0.0 Kafka
(2) Configure the Java Runtime Environment, Kafka boot needs zookeeper, and zookeeper boot requires the Java Runtime environment.
Install good JDK configuration environment variables: java_home, PATH, CLASSPATH
The following can be displayed
[Email protected] bin]# echo $JAVA _home
/usr/local/java/
[Email protected] bin]# echo $CLASSPATH
.:/ Usr/local/java//lib:/usr/local/java//jre/lib
[Email protected] ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/java//bin:/root/bin
(3) Go to the Kafka directory, find the Kafka configuration directory zookeeper configuration file, start zookeeper can use here zookeeper.properties default configuration
Cd/usr/local/kafka/conf
[[email protected] config]# ls zookeeper.properties
Zookeeper.properties

[Email protected] config]# egrep-v ' ^#|^$ ' zookeeper.properties
Datadir=/tmp/zookeeper
clientport=2181
Maxclientcnxns=0
(4) Start the zookeeper with the Kafka script, and note that the script starts with the configuration file. Can be seen from the default configuration file above zookeeper
The default listener port is 2181, which is used to provide consumers. Consumer, the specified socket (localhost+2181), stating that the consumer's message is from zookeeper (coordinated forwarding)
Start Zookeeper:
[Email protected] kafka]#/bin/zookeeper-server-start.sh config/zookeeper.properties

[2016-07-08 21:52:14,446] INFO Reading configuration from:config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2016-07-08 21:52:14,449] INFO Autopurge.snapretaincount set to 3 (Org.apache.zookeeper.server.DatadirCleanupManager)
[2016-07-08 21:52:14,449] INFO Autopurge.purgeinterval set to 0 (Org.apache.zookeeper.server.DatadirCleanupManager)
[2016-07-08 21:52:14,449] INFO Purge task is not scheduled. (Org.apache.zookeeper.server.DatadirCleanupManager)
[2016-07-08 21:52:14,449] WARN either no config or no quorum defined in config, running in standalone mode (Org.apache.zookeeper.server.quorum.Quor Umpeermain)
[2016-07-08 21:52:14,475] INFO Reading configuration from:config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2016-07-08 21:52:14,475] INFO starting server (Org.apache.zookeeper.server.ZooKeeperServerMain)
[2016-07-08 21:52:14,490] INFO Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT ( Org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server Environment:host.name=kafka (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:java.version=1.7.0_71 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:java.vendor=oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server Environment:java.home=/usr/local/java/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:java.class.path=.:/ usr/local/java//lib:/usr/local/java//jre/lib:/usr/local/kafka/bin/. /libs/jetty-security-9.2.12.v20150709.jar:/usr/local/kafka/bin/. /libs/hk2-utils-2.4.0-b31.jar:/usr/local/kafka/bin/. /libs/kafka_2.10-0.9.0.0-scaladoc.jar:/usr/local/kafka/bin/. /libs/zookeeper-3.4.6.jar:/usr/local/kafka/bin/. /libs/slf4j-log4j12-1.7.6.jar:/usr/local/kafka/bin/. /libs/kafka-clients-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/. /libs/kafka_2.10-0.9.0.0-javadoc.jar:/usr/local/kafka/bin/. /libs/kafka_2.10-0.9.0.0-sources.jar:/usr/local/kafka/bin/. /libs/lz4-1.2.0.jar:/usr/local/kafka/bin/. /libs/connect-runtime-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/kafka-log4j-appender-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/snappy-java-1.1.1.7.jar:/usr/local/kafka/bin/. /libs/kafka-tools-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/slf4j-api-1.7.6.jar:/usr/local/kafka/bin/. /libs/jetty-servlet-9.2.12.v20150709.jar:/usr/local/kafka/bin/. /libs/aopalliance-repackaged-2.4.0-b31.jar:/usr/local/kafka/bin/. /libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/. /libs/javax.annotation-api-1.2.jar:/usr/local/kafka/bin/. /libs/jackson-annotations-2.5.0.jar:/usr/local/kafka/bin/. /libs/jetty-io-9.2.12.v20150709.jar:/usr/local/kafka/bin/. /libs/jackson-jaxrs-json-provider-2.5.4.jar:/usr/local/kafka/bin/. /libs/hk2-locator-2.4.0-b31.jar:/usr/local/kafka/bin/. /libs/kafka_2.10-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/jersey-common-2.22.1.jar:/usr/local/kafka/bin/. /libs/jersey-media-jaxb-2.22.1.jar:/usr/local/kafka/bin/. /libs/log4j-1.2.17.jar:/usr/local/kafka/bin/. /libs/jersey-client-2.22.1.jar:/usr/local/kafka/bin/. /libs/jetty-http-9.2.12.v20150709.jar:/usr/local/kafka/bin/. /libs/connect-file-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/jopt-simple-3.2.jar:/usr/local/kafka/bin/. /libs/zkclient-0.7.jar:/usr/local/kafka/bin/. /libs/javax.inject-1.jar:/usr/local/kafka/bin/. /libs/jetty-util-9.2.12.v20150709.jar:/usr/local/kafka/bin/. /libs/jackson-module-jaxb-annotations-2.5.4.jar:/usr/local/kafka/bin/. /libs/scala-library-2.10.5.jar:/usr/local/kafka/bin/. /libs/connect-json-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/connect-api-0.9.0.0.jar:/usr/local/kafka/bin/. /libs/hk2-api-2.4.0-b31.jar:/usr/local/kafka/bin/. /libs/javassist-3.18.1-ga.jar:/usr/local/kafka/bin/. /libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/. /libs/jetty-server-9.2.12.v20150709.jar:/usr/local/kafka/bin/. /libs/argparse4j-0.5.0.jar:/usr/local/kafka/bin/. /libs/jackson-core-2.5.4.jar:/usr/local/kafka/bin/. /libs/jackson-jaxrs-base-2.5.4.jar:/usr/local/kafka/bin/. /libs/jersey-server-2.22.1.jar:/usr/local/kafka/bin/. /libs/javax.ws.rs-api-2.0.1.jar:/usr/local/kafka/bin/. /libs/jersey-guava-2.22.1.jar:/usr/local/kafka/bin/. /libs/jackson-databind-2.5.4.jar:/usr/local/kafka/bin/. /libs/validation-api-1.1.0.final.jar:/usr/local/kafka/bin/. /libs/jersey-container-servlet-2.22.1.jar:/usr/local/kafka/bin/. /libs/jersey-container-servlet-core-2.22.1.jar:/usr/local/kafka/bin/. /libs/javax.inject-2.4.0-b31.jar:/usr/local/kafka/bin/. /libs/kafka_2.10-0.9.0.0-test.jar (Org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server Environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib ( Org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:java.compiler=<na> (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server Environment:os.name=linux (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,490] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,491] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,491] INFO Server environment:user.dir=/usr/local/kafka_2.10-0.9.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,500] INFO Ticktime set to (Org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,500] INFO minsessiontimeout set To-1 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,500] INFO maxsessiontimeout set To-1 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-07-08 21:52:14,511] INFO binding to Port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
Hint message stating that zookeeper has been started can be JPS, netstat authentication process or port with command
[Email protected] ~]# JPS |grep-vi JPS
21380 Quorumpeermain #zookeeper进程
[Email protected] ~]# Netstat-tlnp|grep 2181
TCP 0 0::: 2181:::* LISTEN 21380/java #zookeeper服务端口

(3) Start Kafka with Kafka script, with Kafka configuration file
[Email protected] config]# egrep-v ' ^$|^# ' server.properties
Broker.id=0 #服务id
listeners=plaintext://:9092
port=9092 #kafka默认监听端口
host.name=127.0.0.1 #主机名
Num.network.threads=3
Num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
Log.dirs=/tmp/kafka-logs #kafka的日志文件
Num.partitions=1
Num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
Log.cleaner.enable=false
zookeeper.connect=127.0.0.1:2181 #zookeeper集群
zookeeper.connection.timeout.ms=6000 #zookeeper连接超时时间
Start Kafka:
[Email protected] kafka]#/bin/kafka-server-start.sh config/server.properties
There's a bunch of messages here after startup.
You can also verify the Kafka boot situation with the JPS command or the netstat command.
[Email protected] ~]# JPS |grep-i Kafka
21646 Kafka
[Email protected] ~]# Netstat-tlnp|grep 9092
TCP 0 0::: 9092:::* LISTEN 21646/java
So (3), (4) started zookeeper and Kafka application.
(4) The simulation client sends, accepts the message preliminary test. You must first create a topic
Start Create topic Script:./kafka-topics.sh--create--zookeeper localhost:9092--partitions 1--replication-factor 1--topic kafka_t est02
Explanation, kafka-topics.sh is an automatic topic script for the Kafka installation package, which can be used--help
--create #创建
--zookeeper localhost:9092 #指定--zookeeper provides a service port for consumer.
--partitions 1 #创建一个分区, because topic must have a partition or multiple partitions, create one here.
--replication-factor #副本, a copy of the partition
--replication-factor #指定新创建的topic名
After execution.
[[email protected] kafka]# bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1 --topic kafka-test02
Created topic "kafka-test02". #显示topic kafka-test02 success.
You can also view the existing

[Email protected] kafka]# bin/kafka-topics.sh--list--zookeeper localhost:2181
--topic
Kafak-test
kafka-test02
My_first_topic
(5) Use Kafka to bring a kafka-console-producer.sh to create a product Xshell, with
. bin/kafka-console-producer.sh--broker-list localhost:9092--topic kafka-test02
# kafka-test02 is the topic,producer that was just created, the specified socket (localhost+9092), stating that the producer's message is destined for Kafka, which is also the broker.
[Email protected] kafka]# bin/kafka-console-producer.sh--broker-list localhost:9092--topic kafka-test02
Hello World #在product的shell发送hello World

bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic kafka-test02--from-beginning
#consumer, the specified socket (localhost+2181), stating that the consumer's message came from zookeeper (coordinated forwarding)

[Email protected] kafka]# bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic kafka-test02-- From-beginning

Hello World #在consumer的shell收到hello World information

(5) JPS Command view producer, consumer
[Email protected] ~]# JPS
22380 Consoleproducer
22468 Consoleconsumer
22575 Jps
21646 Kafka
21380 Quorumpeermain
The experiment here is built by a service broker above.

5. Multiple broker clusters of a single node
Start multiple broker services, at which point the Kafka configuration file needs to be re-modified.
[Email protected] config]# egrep-v ' ^$|^# ' server.properties
Broker.id=0 #服务id
listeners=plaintext://:9092
port=9092 #kafka默认监听端口
host.name=127.0.0.1 #主机名
Num.network.threads=3
Num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
Log.dirs=/tmp/kafka-logs #kafka的日志文件
Num.partitions=1
Num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
Log.cleaner.enable=false
zookeeper.connect=127.0.0.1:2181 #zookeeper集群
zookeeper.connection.timeout.ms=6000 #zookeeper连接超时时间

Reference Blog: http://www.aboutyun.com/thread-12847-1-1.html

Kafka experience at the beginning of learning