Build and test the Apache Kafka distributed cluster environment of the message subscription and publishing system

1. What is Kafka?
Kafka is a distributed MQ system developed and open-source by LinkedIn. It is now an incubator project of Apache. On its homepage, Kafka is described as a high-throughput distributed MQ that can distribute messages to different nodes. Kafka is compiled by only 7000 lines of scala. It is understood that Kafka can produce about 0.25 million messages per second (50 MB) and process 0.55 million messages per second (110 MB)

2. Where is the official Kafka website?
Http://kafka.apache.org/

3. Where can I download it? What components are needed?
Kafka2.9.2 can be downloaded at the following address:
Https://www.apache.org/dyn/closer.cgi? Path =/Kafka/0.8.1.1/kafka_2.9.2-0.8.1.1.tgz

4. How to install it?

1. Local Environment Description

Three hosts: centos6.4, Zookeeper 3.4.3, and JDK 1.6 (refer to the previous article)

The IP address and hostname are 192.168.41.100/master, 192.168.41.101/slave1, 192.168.41.102/master, respectively.

2. decompress the kafka_2.9.2-0.8.1.1.tgz and extract it to the/home/hadoop directory.

# tar -zxvf -C /home/hadoop/ kafka_2.9.2-0.8.1.1.tgz

3. Modify the config/server. properties configuration file. The modifications are as follows:

broker.id=0host.name=masteradvertised.host.name=masterzookeeper.connect=master:2181,slave1:2181,slave2:2181

Where

Broker. ID is an integer. We recommend that you set it based on the IP address. Here I use the ID in zookeeper.

Host. Name and advertised. Host. name are the hostnames of the local machine.

Zookeeper. Connect is the connected machine and port.

The modification is as follows:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/49/1A/wKiom1QO8rKDDhKoAABXjFQd4-k786.jpg "style =" float: none; "Title =" 1.png" alt = "wKiom1QO8rKDDhKoAABXjFQd4-k786.jpg"/>

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/49/1B/wKioL1QO8r-D7NuvAAEdY0yUlag626.jpg "style =" float: none; "Title =" 2.png" alt = "wKioL1QO8r-D7NuvAAEdY0yUlag626.jpg"/>

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/49/1A/wKiom1QO8rOxUwdZAAHBpB1BlL0143.jpg "style =" float: none; "Title =" 3.png" alt = "wkiom1qo8roxuwdzaahbpb1bll0143.jpg"/>

4. For bin/Kafka-run-class.shRemove related parameters:

The removal code is

-XX:+UseCompressedOops

Otherwise

Unrecognized VM option ‘UseCompressedOops‘

The modification is as follows:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/49/1B/wKioL1QO8qvCYLubAAC8A321gj8823.jpg "Title =" 4.png" alt = "wkiol1qo8qvcylubaac8a321gj8823.jpg"/>

5. Copy the Kafka folder to another node and modify the content related to server. properties.

#  scp -r /home/hadoop/kafka_2.9.2-0.8.1.1/ slave1:/home/hadoop/kafka_2.9.2-0.8.1.1/

6. Start zookeeper

# zkServer.sh start

After the startup, you can use the following command to view the status on each machine:

# zkServer.sh status

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/49/1B/wKioL1QO8z6SDSbNAACvDNvEJ0c011.jpg "Title =" 5.png" alt = "wkiol1qo8z6sdsbnaacvdnvej0c011.jpg"/>

7. Start Kafka for each node

# kafka-server-start.sh /home/hadoop/kafka_2.9.2-0.8.1.1/config/server.properties &

Part:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/49/1A/wKiom1QO9BfDRZG8AAMI33q1a8c881.jpg "Title =" 6.png" alt = "wkiom1qo9bfdrzg8aami33q1a8c881.jpg"/>

Iv. Simple Installation

1. Create a test topic on the master.

The command is as follows:

# kafka-topics.sh --create --topic test --replication-factor 3 --partitions 2 --zookeeper master:2181

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/49/1A/wKiom1QO9MWx7Y6HAAC3j0VrdJA252.jpg "Title =" 7.png" alt = "wkiom1qo9mwx7y6haac3j0vrdja252.jpg"/>

2. view the test topic on the master.

The command is as follows:

# kafka-topics.sh --list --zookeeper master:2181

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/49/1A/wKiom1QO9S-iG_IUAAB3-mrgr30317.jpg "Title =" 8.png" alt = "wKiom1QO9S-iG_IUAAB3-mrgr30317.jpg"/>

3) send a message to Kafka (slave1 simulated producer) on slave1, send the message "hello", enable a consumer on slave2 (slave2 simulated consumer), and you can see the message just sent

The command on the slave1 host is:

# kafka-console-producer.sh --broker-list master:9092 --sync --topic test

Enter hello

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/49/1B/wKioL1QO9kaxTMKnAAEF9Q3_yBI702.jpg "Title =" 9.png" alt = "wkiol1qo9kaxtmknaaef9q3_ybi702.jpg"/>

After you type Hello:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/49/1A/wKiom1QO9ljRwwGzAAEPTr3xscE724.jpg "Title =" 10.png" alt = "wkiom1qo9ljrwwgzaaeptr3xsce724.jpg"/>

The command on the slave2 host is:

# kafka-console-consumer.sh --zookeeper master:2181 --topic test --from-beginning

The result is as follows:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/49/1A/wKiom1QO9p3jYxAMAADGg8WbYnU114.jpg "Title =" 11.png" alt = "wkiom1qo9p3jyxamaadgg8wbynu114.jpg"/>

This article is from the "xlows" blog, please be sure to keep this source http://xlows.blog.51cto.com/5380484/1550335

Build and test the Apache Kafka distributed cluster environment of the message subscription and publishing system