Resetting the offset of the Kafka topic consumer

If you are using Kafka to distribute messages, there may be exceptions or other errors in the process of data processing that can result in loss or inconsistency. This time you may want to Kafka the data through the new process, we know that Kafka by default will be saved on disk to 7 days of data, you just need to Kafka a topic of the consumer offset to a certain value or a minimum value, You can get the consumer to start spending from the point you set.

Querying the range of topic offset

Use the following command to query to topic:dynamicrange Broker:The minimum value of offset for sparkmaster:9092 :

$/opt/cloudera/parcels/kafka/bin/kafka-run-class. sh kafka.tools.GetOffsetShell--broker-list sparkmaster:9092 -topic dynamicrange-- time- 2

Output

Dynamicrange:0:1288

Query the maximum value of offset:

$/opt/cloudera/parcels/kafka/bin/kafka-run-class. sh kafka.tools.GetOffsetShell--broker-list sparkmaster:9092 -topic dynamicrange-- time- 1

Output

Dynamicrange:0:7885

From the above output can be seen topic:dynamicrange only one partition:0 offset Range:[1288,7885]

Set offset for consumer group

Start Zookeeper Client

$/opt/cloudera/parcels/cdh/lib/zookeeper/bin/zkcli. SH

Set consumer group with the following command:dynamicrangegroup topic:dynamicrange partition:0 with offset of 1288:

set/consumers/dynamicrangegroup/offsets/dynamicrange/0 1288

Reference: Https://metabroadcast.com/blog/resetting-kafka-offsets

Reprint Please specify source: http://www.cnblogs.com/keitsi/p/5685576.html

Resetting the offset of the Kafka topic consumer