Kafka Development Environment Construction (v)

If you want to use code to run Kafka application, then you'd better first give the official website example in a single-machine environment and distributed environment to run, and then gradually replace the original consumer, producer and broker to write their own code. So before reading this article you need to have the following prerequisites:

1. Simple understanding of the Kafka function, understanding the Kafka distributed principle

2. Can run-topic test successfully in distributed environment.

If you have not completed these two prerequisites, please look first:

Kafka distributed preliminary Kafka to build distributed environment

Next, we will briefly introduce the construction of the Kafka development environment.

1. Build the Scala environment, here are two options, the first to go to the Scala official website to download the SDK, unzip the configuration environment variables; the second way, you can not install the SDK, The Scala compilation and encoding packages (Scala-compiler.jar and Scala-library.jar) that are downloaded from Kafka compilation are referenced directly in the project. I recommend the second, because the Scala version and the Kafka version that are downloaded through Kafka compilation are matched (but sometimes may conflict with the environment that Eclipse's plugin needs, so it's best to install the first one, just in case), and generally we use a Java project to write, So it is possible to import the dependent packages directly, and the first scenario helps us to look at the source code and develop it in Scala. The paths to these jars are located in the Kafka-0.7.2-incubating-src\javatest\lib directory.

2. Install the Scala development environment for Eclipse. This is just a plug-in that can be downloaded or installed online in the: http://scala-ide.org/. Once you're done, you'll be able to create a Scala project in Eclipse.

We can write a Hello world first to try, so that there is not another language to write Hello world.

Some people don't find Scala-related classes when they're new, because your eclipse's perspective is not right, so you can switch to Scala's perspective. Note that the new is object and then enter:

?

12345678

package com.a 2 .kafka.scala.test  object hello {      def main (args : array[ String]) : unit = {      printf (    }

3. Find the dependent packages required for the encoding. Remember to go to the Update Kafka folder on your Linux and do not find it from files downloaded directly from the official website. The specific path is: Kafka-0.7.2-incubating-src\javatest\lib This is the most basic package you can use to develop Kafka related programs in Java, if you use Hadoop, just go to the relevant folder to find it. Then add these packages to the project.

Here, the basic development environment should be finished, and then we will start to write some simple code. We are still based on the example given in the previous distributed environment building. A little recollection of:

1. Start Zookeeper server:bin/zookeeper-server-start.sh. /config/zookeeper.properties & (with & to be able to exit the command line)

2. Start Kafka server:bin/kafka-server-start.sh. /config/server.properties &

3. Kafka provides us with a console to do connectivity testing, let's run producer:bin/kafka-console-producer.sh--zookeeper 192.168.10.11:2181 first-- Topic test this is equivalent to opening a producer command line.

4. Next run consumer, city a terminal:bin/kafka-console-consumer.sh--zookeeper 192.168.10.11:2181--topic test-- From-beginning

5. After executing the consumer command, you can enter the information in the producer terminal, and the message will appear in consumer terminal immediately.

This example is to producer production data in a distributed environment and then consumer the data from the broker to display on the console. Of course note a little server.properties in the hostname need to be replaced by your corresponding address, specifically can go back to see the "Distributed environment building." Now we'll use the code to simulate the process of producer sending data:

Here we build a Java project so we can import the dependent packages. The jar under the Kafka-0.7.2-incubating-src\javatest\lib directory.

?

1234567891011121314151617181920

package com.a2.test.kafka;import java.util.Properties;import kafka.javaapi.producer.Producer;import kafka.javaapi.producer.ProducerData; import kafka.producer.ProducerConfig;public class Producertest {    public static void main(String[] args) {        Properties props = new Properties();        props.put("zk.connect", "192.168.10.11:2181");        props.put("serializer.class", "kafka.serializer.StringEncoder");        ProducerConfig config = new ProducerConfig(props);        Producer<String, String> producer = new Producer<String, String>(config);        ProducerData<String, String> data = new ProducerData<String, String>("test", "Hello");        producer.send(data);    }}

So we use the code instead of the console to send the message, here we use the Properties object in the Utils directly to build the configuration, rather than directly read, of course, you can also read the configuration. The two parameters in data, the first one is the specified topic, the second is the content to send.

The next step is to run, if you are under Windows, may wait a long time and then report unable to connect to zookeeper server within timeout:6000, this may be the reason for the NIC, you can put it directly on Linux, Then run with the command line, and note the lead package.

?

1	Java –jar test.jar  Dclasspath=/lib

Consumer code and implementation and producer almost, if you are interested can go to the official website to find the relevant code, are very simple. Of course there is a part of the producer and consumer configuration, we will say next.

Kafka Development Environment Construction (v)