Kafka Distributed Environment Construction (b) likes

This article will show you how to build the Kafka environment, and we'll start with a standalone version and then gradually expand to distributed. Stand-alone version of the building official online, it is easier to achieve, here I will simply introduce the next can, and distributed to build the official website is not described, our ultimate goal is to use distributed to solve the problem, so this part will be the focus.

There are not many Chinese documents in Kafka, so we try to write in detail. To rendezvous you to build the distribution is actually very simple, hands-on tutorial The big deal I recorded a video, but I think that is not the way to go. Only when we really understand the principle and understand thoroughly can we maximize the role of a framework. So, if you don't know anything Kafka, please read first: "Kafka preliminary"

Let's start with setting up a stand-alone version of the environment, if you like the English version: Here is the official "Quick Start"

The deployment of stand-alone version is very simple, I will say a few more important, first Kafka is written in Scala, can run on the JVM, so we do not need to separate to build the Scala environment, will be related to programming when we say how to configure the Scala problem, here is not used, Of course you need to know that this is running on Linux. Second, I use the latest version of 0.7.2, you download the Kafka you can open its directory to browse:

I'm not going to tell you what's in each package, and I'm going to focus on the fact that you can only find 3 jar packages in this folder, and these 3 can't be used for the next programming, and you can't find the XML used to build the POM in it. Also don't worry, don't look for the world, these rely on the library until you put it on Linux to appear (of course, command).

To set up a stand-alone environment, simply say that there are a few steps:

1. Install the Java environment, I use the latest version of JDK7

2. Throw the downloaded Kafka onto Linux and unzip it. I used the red Het server for Linux.

3. The next step is to download the Kafka dependency package and build the Kafka environment. Note that this step requires a computer network. Specific commands are on the official website./SBT Update and./SBT package.

4. The implementation of this step will probably take a more than 10 minutes, I am in my home Ubuntu did not succeed, reported the download is not jline error. Unit with virtual machine Ubuntu success, I deeply doubt is the problem of the net. It's not over. There are two points to note that SBT has helped you to download all the dependent libraries, but these jars are scattered in each directory, note the distinction. Second, these jars are part of the Kafka programming package, which is part of the Scala environment package, which says there's no need to take the Scala environment on your own, right here, you go to the next 2.9 Scala, but people Kafka only support 2.8, 2.7. So when programming, you can use SBT to give you a good package. Later, when it comes to programming, it's easy to write how to build a programming environment.

The above steps are done, the environment is good, below we want to test the success of running Kafka:

1. Start Zookeeper server:bin/zookeeper-server-start.sh. /config/zookeeper.properties & (with & to be able to exit the command line)

2. Start Kafka server:bin/kafka-server-start.sh. /config/server.properties &

3. Kafka provides us with a console for connectivity testing, and we'll run producer:bin/kafka-console-producer.sh--zookeeper localhost:2181--topic Test this is equivalent to opening a producer command line. Parameters of the command line we'll explain it in a moment.

4. Next run consumer, city a terminal:bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic test--from-beginning

5. After executing the consumer command, you can enter the information in the producer terminal, and the message will appear in consumer terminal immediately. Kinda like a communications client.

See "Quick Start" for details

If you can see 5 executed, your standalone deployment is successful. The following explains the meaning of the parameters in the next two commands. --zookeeper localhost:2181 This explains the zookeeper Server,--topic test to connect to the native 2181 port, in Kafka, the message is divided by topic, our topic here is called Test, So both the consumer and the producer point to test. The other parameters, I will, first of all, is the producer parameter:

Parameters of the consumer:

These parameters you can first look at a general, and then in the programming to use, can be dynamically configured.

All right, stand-alone version of the deployment is over, it is not I put consumer on another machine even if distributed. Yes, the premise is that you can still run to the 5th step above. Before we talk about configuration, we'll review the distribution of the previous write, and of course we'll simplify the situation and follow the actual deployment analysis:

Suppose I had only two machines, Server1 and Server2. I now want to put zookeeper server and Kafka server and producer on one machine Server1, put consumer on Server2. This is also called distributed, although the machine is not many, but this deployment has been successful, the extension is quite easy.

We still follow the 5 steps to do, of course, you can certainly know, 3, 42 steps to change the parameters, we assume that Server1 IP is 192.168.10.11 server2 IP is 192.168.10.10:

1. Start Zookeeper server:bin/zookeeper-server-start.sh. /config/zookeeper.properties & (with & to be able to exit the command line)

2. Start Kafka server:bin/kafka-server-start.sh. /config/server.properties &

3. Kafka provides us with a console to do connectivity testing, let's run producer:bin/kafka-console-producer.sh--zookeeper 192.168.10.11:2181 first-- Topic test this is equivalent to opening a producer command line.

4. Next run consumer, city a terminal:bin/kafka-console-consumer.sh--zookeeper 192.168.10.11:2181--topic test-- From-beginning

5. After executing the consumer command, you can enter the information in the producer terminal, and the message will appear in consumer terminal immediately.

This time you can carry out the 5th step effect, is not reported the following error:

I would like to ask you to go back to "Kafka preliminary" and see what is distributed in it:

The Kafka server here is Broker,broker is the data, producer the data to Broker,consumer from the broker to fetch data. That zookeeper is what, said superficial, zookeeper is their choice distributor, all connections must first register to zookeeper. You can think of it as Nio,zookeeper that selector,producer, consumer, and broker all register to selector and leave the corresponding key.

So the problem is on the configuration server.properties of the Kafka server. Kafka registration to the zookeeper on the wrong information, resulting in the above error. Let's look at the configuration of Server.properties in config to know:

?

1234	# Hostname the broker will advertise to consumers. If not set, kafka will use the value returned# from InetAddress.getLocalHost().  If there are multiple interfaces getLocalHost# may not be what you want.#hostname=

Default hostname If you do not set it, it is 127.0.0.1, so you set this hostname to 192.168.10.11, so that you can restart the Kafka server side, you will be able to perform the 5th step.

Successful configuration, you can see the following effect, the left is producer, the right is consumer, see the bottom two good, the front is I used before the test:

If you are still foggy, it is recommended that you go back to the previous article, the Kafka distributed Basic Principles, Kafka the actual operation is to be based on the principle of familiarity with the situation.

After setting up the environment, we will begin to write procedures to deal with the actual problems. Of course, before writing the program, the next I will write a little Kafka why there is such a high performance, how it is to protect these performance.

SOURCE http://my.oschina.net/ielts0909/blog/93190

Kafka Distributed Environment Construction (b) likes