Kafka Getting Started-basic command Operations _kafka

Kafka installation is not introduced, you can refer to the information on the Internet, here mainly introduces the commonly used commands, convenient day-to-day operation and commissioning. Start Kafka

Create topic

bin/kafka-topics.sh--zookeeper **:2181--create--topic * *--partitions--replication-factor 2

Note: The first **IP address, the second * * Theme name partitions the number of partitions replication-factor partition replicas

Number of partitions partitions:

Partitions: partition number, control how many log topic will be fragmented into. Can display the specified, if not specified, the number of num.partitions configurations in the broker (server.properties) will be used

Although increasing the number of partitions can provide Kafka cluster throughput, but too many partitions, or too many partitions on a single server, increases the risk of being unavailable and delayed. Because of the number of partitions, it means that you need to open more file handles, increase point-to-point latency, and increase the client's memory consumption.

The number of partitions also limits the degree of parallelism of the consumer, that is, the number of threads that limit parallel consumer messages cannot be greater than the number of partitions

The number of partitions also limits the producer send message to the specified partition. When the partition is set to 1,producer when the topic is created, it is possible to specify that the partition is 2 or above by a custom partition method; This can be done by alter–partitions to increase the number of partitions.

Replication-factor Copy

Replication factor control messages are stored on several broker (servers) and are generally equal to the number of broker.

The number of Default.replication.factor configurations in broker (server.properties) are used when the specified or through API is not displayed when creating messages to a nonexistent topic production

Query Topic list

bin/kafka-topics.sh--zookeeper **:2181--list

Query Topic Information

bin/kafka-topics.sh--zookeeper **:2181--describe--topic * *

Console to topic Production data

bin/kafka-console-producer.sh--broker-list **:9092--topic * *

Data for console consumption topic

bin/kafka-console-consumer.sh--zookeeper **2181--topic * *--from-beginning

View the maximum (small) value of a topic partition offset

bin/kafka-run-class.sh Kafka.tools.GetOffsetShell--topic hive-mdatabase-hostsltable--time-1--broker-list **:9092- -partitions 0

Note: The maximum value is represented when time is 1, and the minimum value increases the number of topic partitions when time is 2.

bin/kafka-topics.sh--zookeeper node01:2181--alter--topic * *--partitions 10

Delete topic, use caution, only delete metadata in zookeeper, message file must be deleted manually

bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand--zookeeper **:2181--topic * *

Files under logs, with the topic named files, select the number of backups Replication-factor will appear on several nodes view consumption progress

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker--zookeeper **:2181--group * *--topic * *

Where group is the GroupID consumer name. The query method is to query from the zookeeper. The Query method is as follows:

[Hadoop@h71 zookeeper-3.4.5-cdh5.5.2]$ bin/zkcli.sh
[Zk:localhost:2181 (CONNECTED) 2] Ls/consumers