Kafka How to read the offset topic content (__consumer_offsets)

Kafka How to read the offset topic content (__consumer_offsets)

As we all know, since zookeeper is not suitable for frequent write operations in large quantities, the new version Kafka has recommended that consumer's displacement information be kept in topic within Kafka, __consumer_offsets topic, and by default Kafka _consumer_groups.sh script for users to view consumer information.

However, there are still a lot of users who want to know exactly what information is stored inside the __consumer_offsets topic, especially to query how some consumer group's displacements are saved in that topic. To solve these problems, this article will combine an example to explore how to use Kafka-simple-consumer-shell script to query the internal topic.

1. Create topic "Test"

bin/kafka-topics.sh--zookeeper localhost:2181--create--topic Test--replication-factor 3--partitions 3

2. Use kafka-console-producer.sh script to produce messages

Because key is not specified by default, messages are distributed to different partitions according to the Round-robin method. (64 messages are produced in this example)

3. Verify Message Production success

bin/kafka-run-class.sh Kafka.tools.GetOffsetShell--broker-list localhost:9092,localhost:9093,localhost:9094-- Topic Test--time-1

The result output indicates that all 64 messages were successfully produced.

Test:2:21

Test:1:21

Test:0:22

4. Create a console consumer group

bin/kafka-console-consumer.sh--bootstrap-server localhost:9092,localhost:9093,localhost:9094--topic test-- From-beginning--new-consumer

5. Gets the group ID of the consumer group (the following needs to query its displacement information based on that ID)

bin/kafka-consumer-groups.sh--bootstrap-server localhost:9092,localhost:9093,localhost:9094--list--new-consumer

Output: console-consumer-46965 (remember this ID.) )

6. Query __consumer_offsets topic All content

Note: You must set the Exclude.internal.topics=false in Consumer.properties before running the following command

0.11.0.0 Previous version

bin/kafka-console-consumer.sh--topic __consumer_offsets--zookeeper localhost:2181--formatter " Kafka.coordinator.groupmetadatamanager\ $OffsetsMessageFormatter "--consumer.config config/consumer.properties-- From-beginning

After 0.11.0.0 version (included)

bin/kafka-console-consumer.sh--topic __consumer_offsets--zookeeper localhost:2181--formatter " Kafka.coordinator.group.groupmetadatamanager\ $OffsetsMessageFormatter "--consumer.config config/ Consumer.properties--from-beginning

By default __consumer_offsets has 50 partitions, and if you have a lot of consumer group in your system, the output of this command will be much.

7. Compute the partition information of the specified consumer group in __consumer_offsets topic

This is where the 5th-step group.id (console-consumer-46965 in this case) is used. Kafka uses the following formula to calculate which partition the group displacement is saved on __consumer_offsets:

Math.Abs (Groupid.hashcode ())% numpartitions

So in this case, the corresponding partition =math.abs ("console-consumer-46965". Hashcode ())% 50 = 11, which is __consumer_offsets's partition 11, preserves this consumer Group's displacement information, let's verify it below.

8. Get the displacement information for the specified consumer group

Before 0.11.0.0 version

bin/kafka-simple-consumer-shell.sh--topic __consumer_offsets--partition--broker-list localhost:9092,localhost : 9093,localhost:9094--formatter "Kafka.coordinator.groupmetadatamanager\ $OffsetsMessageFormatter"

After 0.11.0.0 version (included)

bin/kafka-simple-consumer-shell.sh--topic __consumer_offsets--partition--broker-list localhost:9092,localhost : 9093,localhost:9094--formatter "Kafka.coordinator.group.groupmetadatamanager\ $OffsetsMessageFormatter"

The following are the output results:

As can be seen above, the consumer group is stored on partition 11 and the Displacement information is right (the displacement information here is the displacement of the consumption, strictly not the displacement in the 3rd step.) Since my consumer has consumed all the information, the displacement here is the same as the displacement in step 3rd. In addition, you can see the format of each log entry for __consumer_offsets Topic: [Group, Topic, Partition]::[offsetmetadata[offset, Metadata], committime , Expirationtime]

Okay, you should already know how to query the contents of __consumer_offsets topic. I hope this article will be of some help to you. (Kafka of course also provides Java APIs for queries, the specific use of the method is not here to repeat, interested can see here.) )