Getting Started with Apache Kafka
In order to facilitate later use, the recording of their own learning process. Because there is no production link use of experience, I hope that experienced friends can leave message guidance.
The introduction of Apache Kafka is probably divided into 5 blogs, the content is basic, the plan contains the following content: Kafka basic configuration and Run Kafka command details kafka-manager basic configuration and running Kafka API simple usage Spring Boot integrated ka Fka
Kafka supports Linux and Windows environments, and this article runs the environment using Linux (CentOS).
This is the first article. Basic configuration and operation of Kafka
Kafka official https://kafka.apache.org
Kafka official documentation is very detailed, the quick Start is also very friendly, although it is in English, but look at the command can understand, Quick start https://kafka.apache.org/quickstart. The following is a basic excerpt from the official QuickStart, suggesting a better understanding of the English version. 1. Download code
First download Kafka from the following address, the current version is 0.10.2.1.
Https://kafka.apache.org/downloads
After the download is complete, unzip the compressed package and enter the Kafka directory:
> TAR-XZF kafka_2.11-0.10.2.1.tgz
> CD kafka_2.11-0.10.2.1
2. Start the server (1) Start zookeeper
Kafka uses zookeeper, so you need to first start a zookeeper server if you haven't. You can use the handy script packaged with Kafka to get a fast but rough single node zookeeper instance.
> bin/zookeeper-server-start.sh config/zookeeper.properties
There are 3 main configurations in this zookeeper:
# The directory where the snapshot is stored.
Datadir=/tmp/zookeeper
# The port at which the clients'll connect
clientport=2181
# Disable the PER-IP limit On the number of connections since a non-production config
maxclientcnxns=0
We need to remember zookeeper Port 2181, which will be used later. (2) Kafka basic Configuration
Kafka provides a basic configuration file in the Config directory. To ensure that Kafka can be accessed remotely, we need to modify both configurations.
Open the Config/server.properties file, in a very forward position with listeners and advertised.listeners two configuration comments, remove the two comments, and according to the current server's IP modified as follows:
# The address of the socket server listens on. It'll get the value returned from
# java.net.InetAddress.getCanonicalHostName () if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = plaintext://your.host.name:9092
listeners=plaintext://:9092
# Hostname and port the broker would advertise to Producers and consumers. If not set,
# It uses the ' value for ' listeners ' if configured. Otherwise, it would use the value
# returned from Java.net.InetAddress.getCanonicalHostName ().
advertised.listeners=plaintext://192.168.16.150:9092
The current server IP is 192.168.16.150 and you need to modify the server IP that is accessible to the extranet or LAN. (3) Start Kafka
Next, start the Kafka service:
> bin/kafka-server-start.sh config/server.properties
If you appear java.lang.OutOfMemoryError:Map at startup failed
Open the bin/kafka-run-class.sh file,
Search-XX:+DISABLEEXPLICITGC,
Replace this parameter with-xx:+explicitgcinvokesconcurrent.
Specific reasons can refer to: http://blog.csdn.net/xieyuooo/article/details/7547435. Create Topic
Use the following command to create the Topic.
> bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test
In addition to the command method, using the Kafka-manager described later can be more convenient to create. Start a consumer
Execute the following command at a new terminal.
> bin/kafka-console-consumer.sh--bootstrap-server localhost:9092--topic test--from-beginning
This consumer is simply outputting messages to the console. Start producer
> bin/kafka-console-producer.sh--broker-list localhost:9092--topic test
Once started, you can enter the content, and then return.
At this point you should be able to see the message output in the previous consumer. The display effect is as follows
The above figure is for the consumer, simply outputting the message to the console. Below is the producer, and the message is generated when carriage returns.
The basic command for Kafka is over, followed by the Kafka command line details.