Kafka Quick Installation Use

Quick Start

This tutorial assumes is starting fresh and has no existing Kafka or ZooKeeper data.

Step 1:download The Code

Download the 0.8.2.0 release and Un-tar it.

> Tar-xzf kafka_2.10-0.8.2.0.tgz> CD kafka_2.10-0.8.2.0

Step 2:start the server

Kafka uses ZooKeeper so, need to first start a ZooKeeper the server if you do not already have one. You can use the convenience script packaged with Kafka to get a quick-and-dirty single-node ZooKeeper instance.

> bin/zookeeper-server-start.sh config/zookeeper.properties[2013-04-22 15:01:37,495] INFO Reading Configuration From:config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) ...

Now start the Kafka server:

> bin/kafka-server-start.sh config/server.properties[2013-04-22 15:01:47,028] INFO Verifying properties ( kafka.utils.VerifiableProperties) [2013-04-22 15:01:47,051] INFO property Socket.send.buffer.bytes was overridden to 1048576 (kafka.utils.VerifiableProperties) ...

Step 3:create a topic

Let's create a topic named "Test" with a single partition and only one replica:

> bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test

We can now see that topic if we run the list topic command:

> bin/kafka-topics.sh--list--zookeeper localhost:2181test

Alternatively, instead of manually creating topics you can also configure your brokers to Auto-create topics when a Non-ex Istent topic is published to.

Step 4:send Some messages

Kafka comes with a command line client that would take input from a file or from the standard input and send it out as messages to the Kafka cluster. By default, line would be sent as a separate message.

Run the producer and then type a few messages into the console to send to the server.

> bin/kafka-console-producer.sh--broker-list localhost:9092--topic Test this was a messagethis is another message

Step 5:start a consumer

Kafka also have a command line consumer, that would dump out messages to standard output.

> bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic Test--from-beginningthis is a messageThis is Anoth ER message

If you had each of the above commands running in a different terminal then you should now being able to type messages into t He producer terminal and see them appear in the consumer terminal.

All of the command line tools has additional options; Running the command with no arguments would display usage information documenting them in more detail.

Step 6:setting up a multi-broker cluster

So far we had been running against a single broker, but that's no fun. For Kafka, a single broker was just a cluster of size one, so nothing much changes other than starting a few more broker in Stances. But just-to-get feel for it, let's expand our cluster to three nodes (still all on our local machine).

First we make a config file for each of the brokers:

> CP config/server.properties config/server-1.properties > CP config/server.properties config/ Server-2.properties

Now edit these new files and set the following properties:

Config/server-1.properties:broker.id=1 port=9093 log.dir=/tmp/kafka-logs-1 Config/server-2.properties:broker . id=2 port=9094 Log.dir=/tmp/kafka-logs-2

the broker.id Property was the unique and permanent name of each node in the cluster. We have to override the port and log directory for only because we is running these all on the same machine and we want to Ke EP The brokers from any trying to register on the same port or overwrite each others data.

We already has Zookeeper and our single node started, so we just need to start the new nodes:

> bin/kafka-server-start.sh config/server-1.properties &...> bin/kafka-server-start.sh config/ Server-2.properties &

Now create a new topic with a replication factor of three:

> bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 3--partitions 1--topic My-replicated-topic

Okay But now, we have a cluster how can we know which broker are doing what? To see that run the "Describe topics" command:

> bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topictopic: My-replicated-topicpartitioncount:1replicationfactor:3configs:topic:my-replicated-topicpartition:0leader: 1replicas:1,2,0isr:1,2,0

Here's an explanation of output. The first line gives a summary of all the partitions, each additional line gives information about one partition. Since we have only one partition for this topic there are only one line.

• "Leader" is the node responsible-reads and writes for the given partition. Each node would be is the leader for a randomly selected portion of the partitions.

• "Replicas" is the list of nodes, replicate the log for this partition regardless of whether they was the leader or Eve N if they is currently alive.

• "ISR" is the set of "In-sync" replicas. This is the subset of the replicas list, which is currently alive and caught-up to the leader.

Note that in my example node 1 are the leader for the only partition of the topic.

We can run the same command on the original topic we created to see where it is:

> bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic testtopic:testpartitioncount : 1replicationfactor:1configs:topic:testpartition:0leader:0replicas:0isr:0

So there is no surprise there-the original topic have no replicas and is on server 0, the only server in we cluster when W e created it.

Let's publish a few messages to our new topic:

> bin/kafka-console-producer.sh--broker-list localhost:9092--topic my-replicated-topic...my test message 1my test Message 2^c

Now let ' s consume these messages:

> bin/kafka-console-consumer.sh--zookeeper localhost:2181--from-beginning--topic my-replicated-topic...my test Message 1my test Message 2^c

Now let's test out fault-tolerance. Broker 1 is acting as the leader so let ' s kill it:

> PS | grep server-1.properties7564 ttys002 0:15.91/system/library/frameworks/javavm.framework/versions/1.6/home/bin/ Java...> kill-9 7564

Leadership have switched to one of the slaves and Node 1 are no longer in the In-sync replica set:

> bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topictopic: My-replicated-topicpartitioncount:1replicationfactor:3configs:topic:my-replicated-topicpartition:0leader: 2replicas:1,2,0isr:2,0

But the messages was still be available to consumption even though the leader that took the writes originally was down:

> bin/kafka-console-consumer.sh--zookeeper localhost:2181--from-beginning--topic my-replicated-topic...my test Message 1my test Message 2^c

Kafka Quick Installation Use