Kafka Shell basic commands (including topic additions and deletions)

The content of this section:

1. Create Kafka Topic
2. View all Topic lists
3. View specified topic information
4. Console to topic Production data
5. Data from the console consumption topic
6. View topic the maximum (small) value of a partition offset
7. Increase the number of topic partitions
8. Delete topic, use caution, only delete metadata in zookeeper, message file must be deleted manually
9. View topic consumption Progress

Create Kafka Topic

bin/kafka-topics.sh--zookeeper node01:2181--create--topic t_cdr--partitions  2--replication-factor

Note: partitions Specifies the number of topic partitions, Replication-factor specifies the number of replicas topic per partition

• partitions Number of partitions:
  • Partitions: Number of partitions, controlling how many logs the topic will fragment into. You can display the specified, if not specified, the number of num.partitions configurations that are used in the broker (server.properties)
  • Although increasing the number of partitions can provide the throughput of a Kafka cluster, an excessive number of partitions or too many partitions on a single server increases the risk of unavailability and latency. Because of the number of partitions, you need to open more file handles, increase the point-to-time latency, and increase the client's memory consumption.
  • The number of partitions also limits the degree of parallelism of consumer, which means that the number of threads that limit parallel consumer messages cannot be greater than the number of partitions
  • The number of partitions also restricts the producer send message to be the specified partition. If the partition is set to 1,producer when the topic is created, a custom partitioning method is used to specify that the partition is 2 or higher, and the number of partitions can be increased by alter–partitions.
• Replication-factor Copy
  • Replication factor control messages are stored on several brokers (servers), typically equal to the number of broker.
  • The number of Default.replication.factor configurations that are used in broker (server.properties) If there is no production message specified at creation time or through API to a nonexistent topic

1. View all Topic lists

bin/kafka-topics.sh--zookeeper node01:2181--list

View specified topic information

bin/kafka-topics.sh--zookeeper node01:2181--describe--topic t_cdr

Console to topic Production data

bin/kafka-console-producer.sh--broker-list node86:9092--topic t_cdr

Data from the console consumption topic

bin/kafka-console-consumer.sh  --zookeeper node01:2181  --topic t_cdr-from-beginning

View topic the maximum (small) value of a partition offset

bin/kafka-run-classsh kafka.tools.GetOffsetShell--topic hive-mdatabase-hostsltable  --time-1-- Broker-list node86:9092--partitions 0

Note: The maximum value when time is-1, and time is-2 indicates the minimum value

Increase the number of topic partitions

Add 10 partitions to topic T_CDR

bin/kafka-topics.sh--zookeeper node01:2181  --alter--topic t_cdr--partitions 10

Delete topic, use caution, only delete metadata in zookeeper, message file must be deleted manually

bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --zookeeper node01:2181 --topic t_cdr

View topic consumption Progress

This will show the consumer group's offset condition, must be--group, do not specify--topic, default to all topic

Displays The:consumer Group, Topic, partitions, Offset, LogSize, Lag, Owner for the specified set of Topics and Consumer Group

$ bin/kafka-run-class. Sh kafka.tools.ConsumerOffsetCheckerrequired argument: [Group] Option Description-------------------broker-Info Print Broker Info--Group Consumer Group.--Help Print this message.--topic comma-separated list of consumer topics (all topicsifabsent). --zkconnect ZooKeeper connect string. (default:localhost:2181) Example,bin/kafka-run-class. SH Kafka.tools.ConsumerOffsetChecker--group Pvgroup Topic Pid Offset logsize Lag Owner PV page_visits 0 21 210 None PV page_visits1 19 190 None PV page_visits2 0 None

Kafka Shell basic commands (including topic additions and deletions)