Kafka cluster deployment

Kafka cluster deployment

1. About kafka

Kafka is a high-throughput distributed publish/subscribe message system that can process all the action flow data of a website with a consumer scale. Such actions (Web browsing, search, and other user actions) are a key factor in many social functions on modern networks. This data is usually solved by processing logs and log aggregation due to throughput requirements. This is a feasible solution for log data and offline analysis systems like Hadoop that require real-time processing. Kafka aims to unify online and offline message processing through the parallel loading mechanism of Hadoop, and also to provide real-time consumption through cluster machines.

For more information about Kafka, see:

2. Preparations

1. Configure the IP addresses of each host. Configure the IP addresses of each host as static IP addresses (ensure that each host can communicate normally. To avoid excessive network transmission, we recommend that you use the same network segment)

2. Modify the host name. All hosts in the Kafka cluster must be modified.

3. Configure host ing. Modify the hosts file and add the ing between host IP addresses and host names.

4. Open the corresponding port. The ports configured in the following documents must be open (or firewall disabled) and root.

5. Ensure that the Zookeeper cluster service can run properly. In fact, as long as the Zookeeper cluster is successfully deployed, the above preparations can basically be completed. For details about Zookeeper deployment, refer:

3. Install Kafka

1. Download The kafka installation package and visit the official Kafka website to download the corresponding version. The version used here is 2.9.2-0.8.1.1.

2. Run the following command to decompress the installation package.

Tar-zxvf kafka_2.9.2-0.8.1.1.tgz

3. modify the configuration file. For simple configuration, you only need to modify the/config/server. properties file.

Vim config/server. properties

Content to be modified:

Broker. id (id of the current server in the cluster, starting from 0); port; host. name (current server host name); zookeeper. connect (connected zookeeper cluster); log. dirs (log storage directory, which must be created in advance ).

Example:

4. Upload the configured kafka to other nodes.

Scp-r kafka node2:/usr/

Note: Do not forget to modify the unique configurations of each node, such as broker. id and host. nam, after uploading.

4. Start and test Kafka

1. Start Zookeeper and run the following command to start Kafka. A message is displayed after the startup is successful.

./Bin/kafka-server-start.sh config/server. properties &

2. Test Kafka. Create topics, producer, and consumer respectively, preferably on different nodes. Enter information on the producer console and check whether the consumer console can receive the message.

Create a topic:

./Bin/kafka-topics.sh-zookeeper node1: 2181, node2: 2181, node3: 2181-topic test-replication-factor 2-partitions 3-create

View topics:

./Bin/kafka-topics.sh-zookeeper node1: 2181, node2: 2181, node3: 2181-list

Create a producer:

./Bin/kafka-console-producer.sh-broker-list node1: 9092, node2: 9092, node3: 9092-topic test

Create a consumer:

./Bin/kafka-console-consumer.sh-zookeeper node1: 2181, node2: 2181, node3: 2181-from-begining-topic test

Test:

Enter information in the producer console to check whether the consumer console can receive the message.

Producer:

Consumer

After the above configuration and tests, Kafka has been initially deployed. You can configure and operate Kafka as needed. For more operations and usage of Kafka, see the official website documentation. Https://cwiki.apache.org/confluence/display/KAFKA/Index

Kafka architecture design of the distributed publish/subscribe message system

Apache Kafka code example

Apache Kafka tutorial notes

Principles and features of Apache kafka (0.8 V)

Kafka deployment and code instance

Introduction to Kafka and establishment of Cluster Environment

For details about Kafka, click here
Kafka: click here

This article permanently updates the link address: