Kafka Learning (1) configuration and simple command usage

Source: Internet
Author: User

Kafka Learning (1) configuration and simple command usage

1. Introduction to related concepts in Kafka is a distributed message middleware implemented by scala. the concepts involved are as follows:

The content transmitted in Kafka is called message. The relationship between topics and messages that are grouped by topic is one-to-many.
We call the message publishing process as producer, that is, the producer generates the <topic-> message> pair and then drops it into the kafka cluster.
The process of processing the corresponding message for the subscription topic is consumer.
A node in a Kafka cluster is called a broker attached graph description under: http://kafka.apache.org/documentation.html#introduction
Ii. configuration of key parameters in Kafka Broker (total cluster node) Configuration

Broker. id: the unique log of the int type. dirs: stores kafka data. The default path is/tmp/kafka-logs port: comsumer connection port zookeeper. conect: the link string of zookeeper. The defined format is hostname1: port1, hostname2: port2, hostname3
Num. partitions: a topic can be divided into multiple paritions pipelines. Messages in each partiions are ordered, but the order in multiple paritions is not guaranteed.
2. Consumer Configuration
Group. id: string type indicates the zookeeper of the consumer process group to which the consumer belongs. connect: hostname1: port1, hostname2: port2 (/chroot/path Unified Data Storage path) zookeeper stores the basic information of comsumers and brokers (including topic and partition) of kafka.
3. configure metadata for Producer. book. list: host1: port1, host2: port2 request. required. acks: 0. data is submitted directly after completion (data may be lost when the server crashes. wait until the server acknowledges the request as successful-1.no messages lost producer. type: Determine whether messages synchronously submit sync serializer. class: kafka. serializer. the serialization class of DefaultEncoder message. The default encoder processing type is byte [].
3. Simple Kafka commands

Step 1: Start the server

Start zookeeper first

> Bin/zookeeper-server-start.sh config/zookeeper. properties)
Start the kafka server.
> bin/kafka-server-start.sh config/server.properties

Step 2: Create a topic

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
View topics
> bin/kafka-topics.sh --list --zookeeper localhost:2181

Step 3: Send some messages

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test This is a messageThis is another message

Step 4: Start a client (consumer)

> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.