Kafka is a distributed data stream platform, which is commonly used as message delivery middleware. This article describes the use of Kafka, with Linux as an example (the Windows system simply changes the following command "bin/" to "bin\windows\", the script extension ". sh" to ". Bat") and is suitable for beginners who have just contacted Kafka and zookeeper. Original Http://kafka.apache.org/documentation#quickstart
1. Download the package and unzip it
Https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
> TAR-XZF kafka_2.11-0.10.1.0.tgz
> CD kafka_2.11-0.10.1.0
2. Start Service
Kafka Zookeeper, you can use it directly to build a zookeeper instance of a single node. You can also install your own configuration zookeeper
> bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading Configuration from:config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) ...
Note: The problem with the folder space issue on Windows may cause the startup zookeeper to fail, I start the error directly
Error: The main class Files\java\jdk1.8.0_51\lib;d:\program could not be found or loaded:
Workaround: Modify the Bin\windows\kafka-run-class.bat file 142 lines, add double quotes to%classpath%:
Set command=%java%%kafka_heap_opts%%kafka_jvm_performance_opts%%kafka_jmx_opts%%KAFKA_LOG4J_OPTS%-cp "%CLASSPATH % "%kafka_opts%%*
Start Kafka Server
> bin/kafka-server-start.sh config/server.properties
[2013-04-22 15:01:47,028] INFO Verifying properties ( kafka.utils.VerifiableProperties)
[2013-04-22 15:01:47,051] INFO property socket.send.buffer.bytes is overridden To 1048576 (kafka.utils.VerifiableProperties) ...
3. Create Topic
Topic name is test,1 block and a copy
> bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test
View topic List
> bin/kafka-topics.sh--list--zookeeper localhost:2181
test
4. Producer Send Message
The Kafka command-line client can send a file or standard input as a message to the Kafka cluster. By default, each row is a separate message.
To run producer when sending messages.
> bin/kafka-console-producer.sh--broker-list localhost:9092--topic Test this are a message this is
another me Ssage
Press CTRL + C to exit message forwarding.
5. Consumer Receive Message
Consumer Subscribe to topic test to receive the above message. The command line runs consumer to display the received message at the terminal:
> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--topic Test--from-beginning This is
a Message This is
another message
You can run producer and consumer separately at two terminals, and more intuitively send messages on the other side and receive messages on the other.
6. Set up multiple agent (multi-broker) clusters
Above we are in an agent to send messages, in fact, a proxy is also a cluster, but a single node. Now let's create two proxy instances to see how they work together.
In the case of only one machine, different ports can be opened to differentiate between different agents.
Copy the Server.properties two copies first:
> CP config/server.properties config/server-1.properties
> CP config/server.properties config/ Server-2.properties
Edit copy:
Config/server-1.properties:
broker.id=1
listeners=plaintext://:9093
log.dir=/tmp/kafka-logs-1
Config/server-2.properties:
broker.id=2
listeners=plaintext://:9094
log.dir=/tmp/kafka-logs-2
The Broker.id attribute must be unique for each node. Using a different log directory is for each node to store its own log files and not overwrite each other.
Start these two nodes:
> bin/kafka-server-start.sh config/server-1.properties &
...
> bin/kafka-server-start.sh config/server-2.properties &
...
Create a new topic with 1 blocks and 3 copies:
> bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 3--partitions 1--topic My-replicated-topic
Execute the describe topics command to see which agent is handling the topic.
> bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topic
: My-replicated-topic partitioncount:1 replicationfactor:3 configs:
topic:my-replicated-topic partition:0 Leader:1 replicas:1,2,0 isr:1,2,0
The first line is a topic overview that describes the topic information. Next each row shows the information for the topic a chunk. Because we have only one piece of topic, so there is only a single line.
Leader: Responsible for the reading and writing of the block, randomly elected nodes;
Replica: Copy the node of the chunking;
ISR: A subset of replica, a candidate set of leader, that is, slave.
In this example, Node 1 is the only chunk of leader My-replicated-topic.
Next we send some messages to My-replicated-topic:
> bin/kafka-console-producer.sh--broker-list localhost:9092--topic my-replicated-topic
...
My test message 1 I
test message 2
^c
Accept messages through consumers:
> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--from-beginning--topic my-replicated-topic
...
My test message 1 I
test message 2
^c
Fault tolerant testing. We kill as Broker 1 of leader:
> PS aux | grep server-1.properties
7564 ttys002 0:15.91/system/library/frameworks/javavm.framework/versions/
1.8/home/bin/java ... > kill-9 7564
Execute Describe topics:
> bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topic
: My-replicated-topic partitioncount:1 replicationfactor:3 configs:
topic:my-replicated-topic partition:0 Leader:2 replicas:1,2,0 isr:2,0
You can see node 20% for the new leader, the spare set ISR has no node 1. Such changes do not affect consumers ' access to data at all:
> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--from-beginning--topic my-replicated-topic
...
My test message 1 I
test message 2
^c
7. Kafka Connect input/output data
sending messages and displaying reception messages only from the terminal is clearly not enough to meet everyone's needs. Kafka provides a tool to connect external systems Kafka connector. Kafka Connector can input a variety of data sources as messages (including files, program output, log), or you can write the received messages to a file.
This example creates a file as an example of a read-write file:
> echo-e "Foo\nbar" > Test.txt
Then run two connectors.
> bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/ Connect-file-sink.properties
Three configuration files:
The first is the Kafka link process, including some generic configurations such as connected broker, data serialization format;
The second and third configuration files each specify a connector, which includes a unique connection name, connection class, and so on. Source Connector read the file and sends each line as a message. Sink connector receives the message from the Kafka and writes the file line-by-row.
Source Connector configuration:
Name=local-file-source
connector.class=filestreamsource
tasks.max=1
file=test.txt
topic= Connect-test
Configuration of sink connector:
Name=local-file-sink
connector.class=filestreamsink
tasks.max=1
file=test.sink.txt
topics= Connect-test
Source created the topic "connect-test", sink subscribed to topic "Connect-test". After two connector are started, the test.sink.txt can be generated under the Kafka root directory to verify that the content is correct:
> Cat test.sink.txt
foo
bar
Of course, you can also read topic connect-test from the end consumer
> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--topic connect-test--from-beginning
{" Schema ": {" type ":" string "," optional ": false}," payload ":" foo "}
{" schema ": {" type ":" string "," optional ": false}," Payload ": Bar"}
...
Add content dynamically to the test.txt, consumers can get the message near real-time.
> echo "Another line" >> test.txt
The additional line is displayed at both the Test.sink.txt and the terminal.