Install Kafka to Windows and write Kafka Java client connections Kafka

Recently want to test the performance of Kafka, toss a lot of genius to Kafka installed to the window. The entire process of installation is provided below, which is absolutely usable and complete, while providing complete Kafka Java client code to communicate with Kafka. Here you have to spit, most of the online articles about how to install Kafka on Windows are either incomplete or Kafka client code is wrong or not based on the 0.8 version. But it must be recalled that this article simply introduces one of the installation methods that may not be the most concise.

1, software preparation

Kafka_2.9.1-0.8.2.1.tgz, need to Kafka website download. Sbt-0.13.9.msi, need to download the SBT website. Git bash online a lot and I'm using Git-1.9.4-preview20140611.exe Eclipse Maven Java 7
2, Kafka environment setting 2.1 Switch to Kafka directoryExtract the kafka_2.9.1-0.8.2.1.tgz first, for example, to D:\sam.lin\software\kafka\kafka_2.9.1-0.8.2.1. Using git bash, switch to the Kafka directory. For example:

Some of the components required by Kafka can be obtained using the SBT Update command, as follows:

2.2 SBT UpdateWhen you hit the ENTER key, an error message appears, "Sbt:no such file or directory," where you need to install SBT and download the URL I have already provided in the "software preparation" above. This article downloads the version of Sbt-0.13.9.msi.
SBT Update command will start downloading components, if the first installation, the process is a long time, patiently waiting for it. When the word [success] appears, the installation component is successful.2.3 SBT PackageThe following figure:

When the word [success] appears, the command execution succeeds.

2.4 SBT sbt-dependencyThe following figure:

When [info] org.scala-sbt:sbt:0.13.8 appears, the command execution succeeds. To this end, the Kafka environment has been built. Next, we need to build zookeeper.
3, Zookeeper environment settingNote that when installing zookeeper, it is impossible to use a shell to install on Windows, at least I tried many times unsuccessfully. In fact, Kafka has provided us with the bat script for installing zookeeper in Windows, which are stored on Bin/window in the Kafka installation directory, as follows:



Note that Git bash cannot be used here because GIT will report a syntax error when it executes the bat file. We switch to window cmd command line.
3.1 Modifying Zookeeper and Kafka configuration files1 Modify the Server.properties file in config directory, modify the Log.dirs=/d/sam.lin/software/kafka/kafka_2.9.1-0.8.2.1/kafka-logs
2 Modify the Log4j.properties file in config directory, modify the Log4j.appender.kafkaappender.file=/d/sam.lin/software/kafka/kafka_ 2.9.1-0.8.2.1/logs/server.log

3) Modify the zookeeper.properties file in config directory, add or modify datadir=/d/sam.lin/software/kafka/kafka_2.9.1-0.8.2.1/data/ Zookeeper
3.2 Using the window cmd command line interface, switch to the/bin/window directory 3.3 Start Zookeeper

Command: Zookeeper-server-start.bat. /.. /config/zookeeper.properties
After successful startup, do not close this cmd command line interface, because close it, the zookeeper process is stopped.

4. Start Kafka broker
Command: Kafka-server-start.bat. /.. /config/server.properties
After successful startup, do not close this cmd command line interface, because close it, the Kafka process is stopped.
5. Create Topic
Command: Kafka-run-class.bat kafka.admin.TopicCommand--create--zookeeper localhost:2181--replication-factor 1-- Partitions 1--topic hellotest
This command creates a topic named "Hellotest".
6, send the messageCommand: Kafka-console-producer.bat--broker-list localhost:9092--topic hellotest
Note that the hellotest here is the topic that was just created. After this command is executed, you can enter the message you want to send directly after the command line.
The Samsamsam in the back is the message that I typed and sent to hellotest this topic. Below we create consumers to consume messages.
7. Receive Message
Command: Kafka-console-consumer.bat--zookeeper localhost:2181--topic hellotest--from-beginning
Similarly, must be specified to hellotest this topic consumer message, I in the producer's CMD command-line interface after the input samsamsam, the message of the CMD command line interface immediately received.

8. Use the Java client provided by Kafka to send and receive messagesAs mentioned above, you can use Kafka's own command-line tools to send and receive messages, and here's a description of sending and receiving messages using Java.
8.1 Sender

Package Com.kafka;

Import java.util.Properties;

Import Kafka.javaapi.producer.Producer;
Import Kafka.producer.KeyedMessage;
Import Kafka.producer.ProducerConfig;

public class Sender {public

  static void Main (string[] args) {
    Properties prop = new properties ();
    Prop.put ("Metadata.broker.list", "127.0.0.1:9092");
    Prop.put ("Serializer.class", "Kafka.serializer.StringEncoder");
    Producerconfig producerconfig = new Producerconfig (prop);
    producer<string, string> Producer = new<string, string> Producer (producerconfig);
    String topic = "Hellotest";
    keyedmessage<string, string> message = new<string, string> keyedmessage (topic, "Sam Hello Test message2"); C13/>producer.send (message);
    Producer.close ();
  }

8.2 Consumer

Code A

Package Com.kafka;
Import Java.util.HashMap;
Import java.util.List;
Import Java.util.Map;
Import java.util.Properties;
Import Java.util.concurrent.ExecutorService;
Import java.util.concurrent.Executors;

Import Java.util.concurrent.TimeUnit;
Import Kafka.consumer.ConsumerConfig;
Import Kafka.consumer.KafkaStream;

Import Kafka.javaapi.consumer.ConsumerConnector;
  public class Consumerdemo {private final consumerconnector consumer;
  Private final String topic;

  Private Executorservice executor; Public Consumerdemo (String zookeeper, String GroupID, String atopic) {consumer = Kafka.consumer.Consumer.createJavaCo
    Nsumerconnector (Consumerprops (Zookeeper, groupid));
  This.topic = atopic;
    public void run (int threads) {map<string, integer> topicmap = new hashmap<string, integer> ();
    Topicmap.put (topic, New Integer (threads)); Map<string, list<kafkastream<byte[], byte[]>>> consumermap = Consumer.createmessagestreams (
    TOPICMAP);List<kafkastream<byte[], byte[]>> streams = consumermap.get (topic);

    Executor = Executors.newfixedthreadpool (threads);

    int numthread = 0;
      For (final Kafkastream stream:streams) {executor.submit (new Consumerdemorun (Stream, numthread));
    numthread++; } private static Consumerconfig Consumerprops (String zookeeper, String GroupID) {Properties Properties = new Properties ();
    Config properties file properties.put ("Zookeeper.connect", zookeeper);
    Properties.put ("Group.id", GroupID);
    Properties.put ("zookeeper.session.timeout.ms", "400");
    Properties.put ("zookeeper.sync.time.ms", "200");
    Properties.put ("auto.commit.interval.ms", "1000");

    Properties.put ("Auto.offset.reset", "smallest");
  return new Consumerconfig (properties);
    public void shutdown () {if (consumer!= null) Consumer.shutdown ();

    if (executor!= null) Executor.shutdown (); try {if (!executor.awaittermination) (5(Timeunit.milliseconds)) {System.out.println ("Timed out waiting for consumer"-Threads down, shut UN
      Cleanly ");
    The catch (Interruptedexception e) {System.out.println ("interrupted during shutdown, exiting Uncleanly");
    } public static void Main (string[] args) {String zookeeper = "localhost:2181";
    String GroupID = "group1";
    String topic = "Hellotest";

    int threads = 1;
    Consumerdemo test = new Consumerdemo (zookeeper, GroupID, topic);

    Test.run (threads);
    try {thread.sleep (10000);
  catch (Interruptedexception IE) {} test.shutdown ();
 }
}

Code two

Package Com.kafka;
Import Kafka.consumer.ConsumerIterator;

Import Kafka.consumer.KafkaStream;
  public class Consumerdemorun implements Runnable {private Kafkastream astream;

  private int athread; Public Consumerdemorun (Kafkastream stream, int thread) {astream = stream;//Set stream from main read Athread = Thread Set thread from main read} public void Run () {consumeriterator<byte[], byte[]> iterator = Astream.ite Rator ();
                                                                    Used to//check  Throughout//the List//continiously while (Iterator.hasnext ())
    {System.out.println ("Thread" + Athread + ":" + New String (Iterator.next (). Message ()));

  } System.out.println ("Shutting down Thread:" + athread);
}
} 

To run the above code, you need to add the following dependencies

                 <dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactid>kafka_2.9.2</ artifactid>
			<version>0.8.2.1</version>
		</dependency>

Summary:Installing Kafka on Windows is troublesome, the installation method provided above is cumbersome, there should be a simpler way. I haven't found it yet.