If you are using Kafka to distribute messages, there may be exceptions or other errors in the process of data processing that can result in loss or inconsistency. This time you may want to Kafka the data through the new process, we know that Kafka by default will be saved on disk to 7 days of data, you just need to Kafka a topic of the consumer offset to a certain value or a minimum value, You can get the consumer to start spending from the point you set.
Querying the range of topic offset
Use the following command to query to topic:dynamicrange Broker:The minimum value of offset for sparkmaster:9092 :
$/opt/cloudera/parcels/kafka/bin/kafka-run-class. sh kafka.tools.GetOffsetShell--broker-list sparkmaster:9092 -topic dynamicrange-- time- 2
Output
Dynamicrange:0:1288
Query the maximum value of offset:
$/opt/cloudera/parcels/kafka/bin/kafka-run-class. sh kafka.tools.GetOffsetShell--broker-list sparkmaster:9092 -topic dynamicrange-- time- 1
Output
Dynamicrange:0:7885
From the above output can be seen topic:dynamicrange only one partition:0 offset Range:[1288,7885]
Set offset for consumer group
Start Zookeeper Client
$/opt/cloudera/parcels/cdh/lib/zookeeper/bin/zkcli. SH
Set consumer group with the following command:dynamicrangegroup topic:dynamicrange partition:0 with offset of 1288:
set/consumers/dynamicrangegroup/offsets/dynamicrange/0 1288
Reference: Https://metabroadcast.com/blog/resetting-kafka-offsets
Reprint Please specify source: http://www.cnblogs.com/keitsi/p/5685576.html
Resetting the offset of the Kafka topic consumer