Kafka Single-Machine deployment

Kafka Single-Machine deployment

Kafka is a high-throughput distributed publish-subscribe messaging system, Kafka is a distributed message queue for log processing by LinkedIn, with large log data capacity but low reliability requirements, and its log data mainly includes user behavior

Environment configuration: CentOS Release 6.3 (Final) JDK version: Jdk-6u31-linux-x64-rpm.binzookeeper version: zookeeper-3.4.4

1. Install the JDK and set environment variables

JDK environment variable configuration:

Export Java_home=/usr/java/jdk1.6.0_31export classpath= $JAVA _home/jre/lib: $JAVA _home/libexport PATH= $PATH: $JAVA _ Home/bin

# Source/etc/profile Make it effective

2, Installation Zookeeper

# Vim Conf/zoo.cfg

Datadir=/home/hadoop/storage/zookeeper setting DataDir

Start Zookeeper:

#./bin/zkserver.sh Start

View state of ZK:

#./BIN/ZKSERVER.SH Status

3, Kafka installation

• Modify config file: # vim Config/producer.properties

broker.id=0port=9092host.name=10.37.129.3num.network.threads=2num.io.threads=8socket.send.buffer.bytes= 1048576socket.request.max.bytes=104857600log.dirs=/tmp/kafka-logsnum.partitions=2log.flush.interval.messages= 10000log.flush.interval.ms=1000log.retention.hours=168log.retention.bytes=1073741824log.segment.bytes= 536870912log.retention.check.interval.ms=60000log.cleaner.enable=falsezookeeper.connect=10.37.129.3:2181 zookeeper.connection.timeout.ms=1000000

• Start Kafka:

[Email protected] kafka_2.8.0-0.8.0]#./bin/kafka-server-start.sh/usr/local/src/kafka_2.8.0-0.8.0/config/ Server.properties &

• Create a new topic

#./kafka-topics.sh--create--topic kafkatopic--replication-factor 1--partitions 1--zookeeper localhost:2181

• Check whether the theme was created successfully

#./kafka-list-topic.sh--zookeeper localhost:2181
Topic:test partition:0 leader:0 replicas:0 isr:0

• Send Message to Kafka:

[Email protected] bin]#/kafka-console-producer.sh--broker-list 192.168.1.141:9092--sync--topic kafkatopic

This is the first message

This is another message.

This is another news

Another terminal that shows the consumption of messages

[Email protected] bin]#/kafka-console-consumer.sh--zookeeper 192.168.1.141:2181--topic kafkatopic-- From-beginning

This is the first message

This is another message.

This is another news

* Start Produce
./bin/kafka-console-producer.sh--broker-list 10.10.224.12:9092--topic test
* Start Consumer
./kafka-console-consumer.sh--zookeeper localhost:2181--topic test
6. Close Kafka and Zookeeper
./kafka-server-stop.sh. /config/server.properties
./zookeeper-server-stop.sh
Summary of Experience:
1.produce start when the parameters are used Kafka port and consumer start using the Zookeeper port;
2. Topic must be created before use;
The essence of 3.topic is stored on zookeeper in the form of documents.

This article is from the "david0512" blog, make sure to keep this source http://gjr0512.blog.51cto.com/6518687/1591741

Kafka Single-Machine deployment