Kafka Quick Start

Kafka is a distributed data stream platform, which is commonly used as message delivery middleware. This article describes the use of Kafka, with Linux as an example (the Windows system simply changes the following command "bin/" to "bin\windows\", the script extension ". sh" to ". Bat") and is suitable for beginners who have just contacted Kafka and zookeeper. Original Http://kafka.apache.org/documentation#quickstart

1. Download the package and unzip it

Https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz

> TAR-XZF kafka_2.11-0.10.1.0.tgz
> CD kafka_2.11-0.10.1.0

2. Start Service

Kafka Zookeeper, you can use it directly to build a zookeeper instance of a single node. You can also install your own configuration zookeeper

> bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading Configuration from:config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) ...

Note: The problem with the folder space issue on Windows may cause the startup zookeeper to fail, I start the error directly

Error: The main class Files\java\jdk1.8.0_51\lib;d:\program could not be found or loaded:

Workaround: Modify the Bin\windows\kafka-run-class.bat file 142 lines, add double quotes to%classpath%:

Set command=%java%%kafka_heap_opts%%kafka_jvm_performance_opts%%kafka_jmx_opts%%KAFKA_LOG4J_OPTS%-cp "%CLASSPATH % "%kafka_opts%%*

Start Kafka Server

> bin/kafka-server-start.sh config/server.properties
[2013-04-22 15:01:47,028] INFO Verifying properties ( kafka.utils.VerifiableProperties)
[2013-04-22 15:01:47,051] INFO property socket.send.buffer.bytes is overridden To 1048576 (kafka.utils.VerifiableProperties) ...

3. Create Topic

Topic name is test,1 block and a copy

> bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test

View topic List

> bin/kafka-topics.sh--list--zookeeper localhost:2181
test

4. Producer Send Message

The Kafka command-line client can send a file or standard input as a message to the Kafka cluster. By default, each row is a separate message.

To run producer when sending messages.

> bin/kafka-console-producer.sh--broker-list localhost:9092--topic Test this are a message this is
another me Ssage

Press CTRL + C to exit message forwarding.

5. Consumer Receive Message

Consumer Subscribe to topic test to receive the above message. The command line runs consumer to display the received message at the terminal:

> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--topic Test--from-beginning This is
a Message This is
another message

You can run producer and consumer separately at two terminals, and more intuitively send messages on the other side and receive messages on the other.

6. Set up multiple agent (multi-broker) clusters

Above we are in an agent to send messages, in fact, a proxy is also a cluster, but a single node. Now let's create two proxy instances to see how they work together.

In the case of only one machine, different ports can be opened to differentiate between different agents.

Copy the Server.properties two copies first:

> CP config/server.properties config/server-1.properties
> CP config/server.properties config/ Server-2.properties

Edit copy:

Config/server-1.properties:
    broker.id=1
    listeners=plaintext://:9093
    log.dir=/tmp/kafka-logs-1

Config/server-2.properties:
    broker.id=2
    listeners=plaintext://:9094
    log.dir=/tmp/kafka-logs-2

The Broker.id attribute must be unique for each node. Using a different log directory is for each node to store its own log files and not overwrite each other.

Start these two nodes:

> bin/kafka-server-start.sh config/server-1.properties &
...
> bin/kafka-server-start.sh config/server-2.properties &
...

Create a new topic with 1 blocks and 3 copies:

> bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 3--partitions 1--topic My-replicated-topic

Execute the describe topics command to see which agent is handling the topic.

> bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topic
: My-replicated-topic	partitioncount:1	replicationfactor:3	configs:
	topic:my-replicated-topic	partition:0	Leader:1	replicas:1,2,0	isr:1,2,0

The first line is a topic overview that describes the topic information. Next each row shows the information for the topic a chunk. Because we have only one piece of topic, so there is only a single line.

Leader: Responsible for the reading and writing of the block, randomly elected nodes;

Replica: Copy the node of the chunking;

ISR: A subset of replica, a candidate set of leader, that is, slave.

In this example, Node 1 is the only chunk of leader My-replicated-topic.

Next we send some messages to My-replicated-topic:

> bin/kafka-console-producer.sh--broker-list localhost:9092--topic my-replicated-topic
...
My test message 1 I
test message 2
^c

Accept messages through consumers:

> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--from-beginning--topic my-replicated-topic
...
My test message 1 I
test message 2
^c

Fault tolerant testing. We kill as Broker 1 of leader:

> PS aux | grep server-1.properties
7564  ttys002    0:15.91/system/library/frameworks/javavm.framework/versions/
1.8/home/bin/java ... > kill-9 7564

Execute Describe topics:

> bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topic
: My-replicated-topic	partitioncount:1	replicationfactor:3	configs:
	topic:my-replicated-topic	partition:0	Leader:2	replicas:1,2,0	isr:2,0

You can see node 20% for the new leader, the spare set ISR has no node 1. Such changes do not affect consumers ' access to data at all:

> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--from-beginning--topic my-replicated-topic
...
My test message 1 I
test message 2
^c

7. Kafka Connect input/output data

sending messages and displaying reception messages only from the terminal is clearly not enough to meet everyone's needs. Kafka provides a tool to connect external systems Kafka connector. Kafka Connector can input a variety of data sources as messages (including files, program output, log), or you can write the received messages to a file.

This example creates a file as an example of a read-write file:

> echo-e "Foo\nbar" > Test.txt

Then run two connectors.

> bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/ Connect-file-sink.properties

Three configuration files:

The first is the Kafka link process, including some generic configurations such as connected broker, data serialization format;

The second and third configuration files each specify a connector, which includes a unique connection name, connection class, and so on. Source Connector read the file and sends each line as a message. Sink connector receives the message from the Kafka and writes the file line-by-row.

Source Connector configuration:

Name=local-file-source
connector.class=filestreamsource
tasks.max=1
file=test.txt
topic= Connect-test

Configuration of sink connector:

Name=local-file-sink
connector.class=filestreamsink
tasks.max=1
file=test.sink.txt
topics= Connect-test

Source created the topic "connect-test", sink subscribed to topic "Connect-test". After two connector are started, the test.sink.txt can be generated under the Kafka root directory to verify that the content is correct:

> Cat test.sink.txt
foo
bar

Of course, you can also read topic connect-test from the end consumer

> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--topic connect-test--from-beginning
{" Schema ": {" type ":" string "," optional ": false}," payload ":" foo "}
{" schema ": {" type ":" string "," optional ": false}," Payload ": Bar"}
...

Add content dynamically to the test.txt, consumers can get the message near real-time.

> echo "Another line" >> test.txt

The additional line is displayed at both the Test.sink.txt and the terminal.