Topic operation of Kafka

Kafka Shell Topic operation
Create topic
Hadoop
kafka]# bin/kafka-topics.sh--create--topic hadoop--zookeeper master:2181,slave01:2181,slave02:2181--partitions 1-- Replication-factor 1
kafka]# bin/kafka-topics.sh--create--topic hive--zookeeper master:2181,slave01:2181,slave02:2181--partitions 1-- Replication-factor 1
kafka]# bin/kafka-topics.sh--create--topic hbase--zookeeper master:2181,slave01:2181,slave02:2181--partitions 3-- Replication-factor 1

The problem of creating topic process, the number of replication-factor can not exceed the number of broker
bin/kafka-topics.sh--create--topic sqoop--zookeeper master:2181,slave01:2181,slave02:2181--partitions 3-- Replication-factor 3
Error while executing topic command:replication factor:3 larger than available brokers:1

View topic List
kafka]# bin/kafka-topics.sh--list--zookeeper master:2181,slave01:2181,slave02:2181
To view a specific topic
kafka]# bin/kafka-topics.sh--describe--topic Hadoop--zookeeper master:2181,slave01:2181,slave02:2181

Topic:hadoop partitioncount:1 replicationfactor:1
Configs:
Topic:hadoop partition:0 leader:0 replicas:0 isr:0

The number of partition corresponding to the Partitioncount:topic
Replicationfactor:topic corresponds to the copy factor, White is the number of replicas
Partition:partition number, increment starting from 0
Leader: The current partition works Breaker.id
Replicas: The current replica data sits in the Breaker.id, is a list of the top-most of its functions
ISR: List of breaker.id available in the current KAKFA cluster
Modify Topic
Cannot modify Replication-factor, and can only increase the number of partition, can not reduce
bin/kafka-topics.sh--alter--topic Hive--zookeeper master:2181,slave01:2181,slave02:2181--partitions 3

The exception thrown when the partition is changed from 3 to 2:
ERROR kafka.admin.AdminOperationException:The number of partitions for a topic can only be increased
Delete Topic
kafka]# bin/kafka-topics.sh--delete--topic hbase--zookeeper master:2181,slave01:2181,slave02:2181
Topic HBase is marked for deletion.
Note:this would have no impact if delete.topic.enable are not set to true.
Completely remove a topic, need to configure delete.topic.enable=true in server.properties, otherwise just mark delete

After the configuration is complete, you need to restart the Kafka service

--------------------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------- ------

A simple production and consumption model
Use standard production consumption scripts provided by Kafka
Production data
kafka]# bin/kafka-console-producer.sh--broker-list master:9092--topic Hadoop
Production data needs to be specified: which broker the current data flows to, and which topic
Consumer data
bin/kafka-console-consumer.sh--topic Hadoop--zookeeper master:2181,slave01:2181,slave02:2181
Description: The consumption statement, only get the latest data, to want historical data, need to add options--from-beginning
bin/kafka-console-consumer.sh--topic Hadoop--zookeeper master:2181,slave01:2181,slave02:2181--from-beginning
When consuming data, you only need to specify topic and topic metadata information (stored in ZK), so use ZK

Consumer-blacklist (blacklist) and whitelist (whitelist) options
--blacklist is followed by a list of topic that need to be filtered, separated by "," meaning that other topic data can be received in addition to the topic in the list
--whitelist is followed by a list of topic that need to be filtered, separated by "," meaning that no other topic data can be received except the topic in the list
eg.
bin/kafka-console-consumer.sh--zookeeper master:2181,slave01:2181,slave02:2181--from-beginning--blacklist Hadoop , Hive
bin/kafka-console-consumer.sh--zookeeper master:2181,slave01:2181,slave02:2181--from-beginning--whitelist Hadoop , FL