Kafka Combat-kafkaoffsetmonitor

1. Overview

The background of Kafka and some application scenarios are presented, along with a simple example demonstrating the Kafka. Then, in the process of development, we will find some problems, that is the information monitoring situation. Although, after initiating the related service of Kafka, we produce the message and the consumer message will display these record information in the terminal console, but this is always not friendly, and, in the actual development, we do not have the authority to watch the terminal console, so today we will introduce a monitoring system of Kafka- Kafkaoffsetmonitor. Here are the contents of the catalog that we share today:

• Kafkaoffsetmonitor Brief Introduction
• Kafkaoffsetmonitor Installation Deployment
• Kafkaoffsetmonitor Run Preview

Let's start today's content sharing.

2.KafkaOffsetMonitor Overview

Kafkaoffsetmonitor is a Web management interface provided by the Kafka Open source community, which is used to monitor the consumer of Kafka services and the offset in their partition in real time, and you can browse through the current consumer group , and all the partition consumption of each topic can be seen in full view. It makes us very intuitive to know that each partition message is consumed, there is a block of wood and so on.

This web management platform retains the relevant historical data of partition, offset and its consumer, and we can clearly know the consumption situation in the recent period by browsing the relevant modules of Web management.

The Web management platform has the following features:

• Consumer monitoring of consumer, and list of offset data for each consumer
• Protect consumer group list information
• All partition lists for each Topic include: Topic, Pid, Offset, LogSize, lag, owner, etc.
• Browse topic Historical consumer Information

These features are more than sufficient for our development.

3.KafkaOffsetMonitor Installation Deployment 3.1 Downloads

When installing the Kafkaoffsetmonitor management platform, we need to download its installation package first, its resources can be found on GitHub, considering the limitations of GitHub access, I upload the installation package to Baidu Cloud disk:

《》

3.2 Installation Deployment

Kafkaoffsetmonitor installation deployment is simple, all the resources are packaged into a jar file, so directly run, save us to configure. Here we can create a new directory to be used for Kafka's monitoring directory alone, I create a new Kafka_monitor file directory here, and then we are ready to launch the script, the script content is as follows:

#! /bin/-cp kafkaoffsetmonitor-assembly-0.2. 0 . Jar  com.quantifind.kafka.offsetapp.OffsetGetterWeb  --zk dn1:2181, dn2:2181, DN3:2181  8089  seconds  1 . Days

To explain the meaning of this startup command, first we need to indicate the class that is running Web monitoring, then we need to use zookeeper, all the ZK cluster information to fill in, then the web run port, the time of page data refresh and the time value of preserving the data.

3.3 Start

Next, we start the boot process as follows:

• Step 1: Start ZK (dn1~dn3 node)

Zkserver. SH start

• Step 2: Start the Kafka service (the cluster starts by entering the following command sequentially)

Kafka-server-start. SH

• Step 3: Start the Web monitoring service

Java-CP kafkaoffsetmonitor-assembly-0.2. 0 . Jar  com.quantifind.kafka.offsetapp.OffsetGetterWeb  --zk dn1:2181, dn2:2181 , Dn3:2181  8089  . Seconds   1. Days

After the Web service starts successfully, as shown in:

4.KafkaOffsetMonitor Run Preview

Below, let's use Kafka code production to consume some messages, using web monitoring to browse the message situation. Production code you can refer to the previous I wrote "Kafka combat-simple example", here directly preview the results, as shown in:

5. Summary

When running the Kafkaoffsetmonitor jar package, you need to make sure that the startup parameters are configured correctly to avoid startup errors, and that some links to the static resources in the Kafkaoffsetmonitor jar on GitHub are linked to Google hyperlinks. All if direct use only, if the local wood agent software will start the error, here to use the jar I provided, this jar is a static resource revision after the use of local static resources re-compiled.

In addition, some of the parameters in the diagram have the following meanings:

• Topic: Create Topic Name
• Partition: Partition number
• Offset: Indicates how much message the parition has consumed.
• LogSize: Indicates how many message the partition produced
• Lag: Indicates how many message lines have not been consumed
• Owner: Represents the Consumer
• Created: Indicates the partition creation time
• Last Seen: Indicates consumption status refresh latest time

6. Concluding remarks

This blog is to share with you here, if you study in the process of learning what is the problem, you can add groups to discuss or send e-mail to me, I will do my best to answer for you, with June encouragement!

Kafka Combat-kafkaoffsetmonitor