Kafka Quick Start

1.3 Quick Start Step 1: Download Kafka Click here to download Download and unzip

Tar-xzf kafka_2.10-0.8.2.0.tgz CD kafka_2.10-0.8.2.0

Step 2: Start the service

Kafka uses ZooKeeper so you need to start the ZooKeeper service first. If you do not have a ZooKeeper service, you can use Kafka to bring your own script to launch an emergency single-point ZooKeeper instance.

Bin/zookeeper-server-start.sh config/zookeeper.properties

[2013-04-22 15:01:37,495] INFO Reading configuration from:config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) ...

Now start the Kafka server:

Bin/kafka-server-start.sh config/server.properties

[2013-04-22 15:01:47,028] Info Verifying properties (kafka.utils.VerifiableProperties) [2013-04-22 15:01:47,051] Info Property Socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties) ...

Step 3: Create a topic Let's create a name is Test of the topic, this topic has only one partition and one copy:

bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test

run the following command to view this topic information:

Test

We can also configure , let brokers the topic is created automatically, When a message is posted to a topic that does not exist. Step 4: Send some messages Kafka has a command-line client that can send messages to the Kafka cluster from a file or standard input. The default per row is a separate message.

run producer and enter a few messages to the console, and enter the input to send to the server.

bin/kafka-console-producer.sh--broker-list localhost:9092--topic test

This is a message

This is another message

Step 5: Start a consumer Kafka also has a Consumer   command-line tools can accept messages to standard output.

bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic test--from-beginning

This is a message

This is another message

The run without parameters command displays a detailed usage document.

Step 6: Set up a multi-broker cluster At present we have been in a single Broker on run, it doesn't mean anything. For Kafka, a broker is a cluster that is only 1 in size, so launching multiple broker instances makes no difference. But just-get feel for it, let's expand our cluster to three nodes (all operations still operate on this machine).

first, we are for each Broker prepares a configuration file:

CP Config/server.properties Config/server-1.properties

CP Config/server.properties Config/server-2.properties

Now edit these new files with the following content:

Log.dir=/tmp/kafka-logs-2

each Broker.id property is unique and never changes. Modify Port and the log directory only because all the The broker runs on the same machine, and the production environment recommends a unified port and log directory.

It has been opened before zookeeper and broker, now just need to start two more:

bin/kafka-server-start.sh config/server-1.properties &...

bin/kafka-server-start.sh config/server-2.properties &...

Create a new topic and set the replica factor for it:

bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 3--partitions 1--topic My-replicated-topic

OK, now that we've built the good one cluster, but we don't know how he works. Use the "Describe topics" command to view:

bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topic

Topic:my-replicated-topic partitioncount:1        Replicationfactor:3     

Topic:my-replicated-topic      partition:0    leader:1       replicas:1,2,0 isr:1,2,0

The first line of the output is an overview of all partitions, and each additional row is a description of each partition. Because we only have one partition, so there's only a row here.

• "Leader" is responsible for reading and writing the partition. Each node would be is the leader for a randomly selected portion of the partitions.
• "Replicas" saves a list of nodes for that partition replica.
• "ISR" is the set of "In-sync" replicas. This is the subset of the replicas list, which is currently alive and caught-up to the leader.

Note: Node 1 in this example is the leader of this topic.

we'll look at the original topic:

bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic test

Topic:test     partitioncount:1        replicationfactor:1     

Topic:test     partition:0    leader:0       replicas:0     isr:0

It's no surprise-the original topic didn't set a copy, so Replicas is 0, when we created it, there was only one node in the cluster.

Let's post a few messages to our new topic:

bin/kafka-console-producer.sh--broker-list localhost:9092--topic my-replicated-topic

...

My test message 1

My test message 2

^c

Now let ' s consume these messages:

bin/kafka-console-consumer.sh--zookeeper localhost:2181--from-beginning--topic my-replicated-topic

...

My test message 1

My test message 2

^c

now let's test the fault tolerance. Here Broker 1 is leader so we kill it:

PS | grep server-1.properties

7564 ttys002    0:15.91/system/library/frameworks/javavm.framework/versions/1.6/home/bin/java

...

Kill-9 7564

then leader switches to the other nodes and Node 1 is no longer in-sync set up:

bin/kafka-topics.sh--describe--zookeeper localhost:2181--topic my-replicated-topic

Topic:my-replicated-topic      partitioncount:1        replicationfactor:3     configs:

Topic:my-replicated-topic      partition:0    leader:2       replicas:1,2,0 isr:2,0

but the message can still be consumed, even if the previous leader wrote the log:

bin/kafka-console-consumer.sh--zookeeper localhost:2181--from-beginning--topic my-replicated-topic

...

My test message 1

My test message 2

^c

Kafka Quick Start