Windows installation runs Kafka

Brief introduction

This article describes how to configure and launch Apache Kafka on Windows OS, which will guide you through the installation of Java and Apache Zookeeper.
Apache Kafka is a fast and extensible message queue that can handle heavy read-write workloads, such as IO-related work. For more information, see http://kafka.apache.org. Because zookeeper can provide reliable distributed coordination services, Apache Kafka needs to run a zookeeper instance. For more information about zookeeper, please see https://zookeeper.apache.org/.

For specific steps on installing Kafka in Windows, you can view this video: Https://youtu.be/OJKesEpO6ok

Download the required files

• Download the server JRE http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html here, depending on the system OS and CPU architecture
• Download and install the 7-zip http://www.7-zip.org/download.html here
• Download the zookeeper here and unzip it with 7-zip http://zookeeper.apache.org/releases.html
• Download Kafka here and unzip the http://kafka.apache.org/downloads.html with 7-zip

For this tutorial, we extracted zookeeper and Kafka to the C drive, but can also choose a different location. Here we are going to use the full zookeeper instead of the one packaged with Kafka, because this is a single-node zookeeper instance. You can also run Kafka packaged with zookeeper, located in the \kafka\bin\windows library.

Installation

A. JDK Installation

1. Start the JRE installation, select the checkbox "Modify the Destination path" and click Install.

2. Modify the installation directory, the folder name must not have spaces, for example: C:\Java\jre1.8.0_xx\ (by default, C:\Program files\java\jre1.8.0_xx), and then click Next.
3. Now click the environment variable, advanced system settings, System, Control Panel, open the System Environment Variables dialog box.
4. Click the new User variable button in the user variable, then enter Java_home in the variable name and fill in the variable value with the custom JRE path. As shown in the following:

The Java path and version may vary depending on the version of the Kafka used

5. Click OK now.
6. The Environment Variables dialog box that you just opened has the System Variables column, where you look for path variables.
7. Edit the path and type ";%java_home%\bin", such as:

8. Confirm that the Java installation opens CMD, enter the type "Java–version", you should be able to see the Java version just installed.

If the command line prompt is similar, continue. Otherwise, you will need to recheck whether the installation version matches the OS schema (x86, x64), and whether the environment variable path is correct.

B. Zookeeper Installation

1. Enter the Zookeeper settings directory, such as C:\zookeeper-3.4.7\conf
2. Rename "Zoo_sample.cfg" to "zoo.cfg".
3. Open Zoo.cfg in any text editor (such as Notepad), I personally prefer notepad++.
4. Locate and edit Datadir=/tmp/zookeeper to: \zookeeper-3.4.7\data
5. Similar to what is done in Java, we add the following in the system environment variable:
A. Adding Zookeeper_home = C:\zookeeper-3.4.7 in System variables
B. Edit system variables, named path system variable%zookeeper_home%\bin;
6. Modify the default zookeeper port (default port 2181) in the Zoo.cfg file.
7. Open the new cmd, enter Zkserver, and run zookeeper.
8. Command-line prompts are as follows:

Congratulations, zookeeper is complete and running on port 2181.

C. Installing Kafka

1. Enter the Kafka configuration directory, e.g. C:\kafka_2.11-0.9.0.0\config
2. Edit the file "Server.properties"
3. Find and edit "Log.dirs=/tmp/kafka-logs" to "log.dir= C:\kafka_2.11-0.9.0.0\kafka-logs"
4. If zookeeper is running on some other machine or cluster, you can modify "zookeeper.connect:2181" to custom IP and port. We used the same machine in this demo, so there's no need to make any changes. The Kafka ports and Broker.id in the file are also configurable. The other settings do not change.
5. Kafka will run by default on port 9092 and connect to Zookeeper's default port: 2181.

D. Running the Kafka server

Important: Make sure that the zookeeper instance is ready and running before you start the Kafka server.

1. Enter the Kafka installation directory C:\kafka_2.11-0.9.0.0\
2. Press shift+ Right-click and select the "Open command Window" option to open the command line.
3. Now enter. \bin\windows\kafka-server-start.bat. \config\server.properties and enter.

\bin  \windows  \kafka -server-start.bat. \config  \server . Properties  

4. If everything is OK, the command line should be:

5. Now that the Kafka is ready and running, you can create a theme to store the message. We can also generate or consume data from the Java/scala code, or directly from the command line.

E. Creating a Theme

1. Now create a theme, named "Test", Replication factor=1 (because only 1 Kafka servers are running). If there are more than 1 Kafka servers running in the cluster, you can increase the replication-factor accordingly, which increases data availability and system fault tolerance.
2. Open a new command line in C:\kafka_2.11-0.9.0.0\bin\windows.
3. Enter the following command, carriage return:

Kafka-Topics.Bat --Create --Zookeeper localhost:2181 --Replication-Factor 1 --Partitions 1 --Topic Test

F. Create producer and consumer to test the server.

1. Open a new command line in C:\kafka_2.11-0.9.0.0\bin\windows.
2. Enter the following command to start producer:

kafka-console  -producer   Bat -- broker- List  localhost:9092  -- topic test  

3. In the same location C:\kafka_2.11-0.9.0.0\bin\windows open the new command line again.
4. Now enter the following command to start consumer:

Kafka-Console-Consumer.Bat --Zookeeper localhost:2181 --Topic Test

5. There are now two command-line windows, such as:

6. Enter any content in the producer command line, and you can see the message in the other consumer command line.

7. If you can push the message to the consumer side and display it, the Kafka installation is complete.

A few useful commands

1. List of topics: Kafka-topics.bat–list–zookeeper localhost:2181
2. Description topic: Kafka-topics.bat–describe–zookeeper localhost:2181–topic [Topic Name]
3. Read the message from the beginning: kafka-console-consumer.bat–zookeeper localhost:2181–topic [Topic name]–from-beginning
4. Delete topic: Kafka-run-class.bat kafka.admin.topiccommand–delete–topic [Topic_to_delete]–zookeeper localhost:2181

Windows installation runs Kafka