Apache Kafka Surveillance Series-kafkaoffsetmonitor

Original link:Apache Kafka Surveillance series-kafkaoffsetmonitorOverview

Recently the Kafka server messaging service was online, and the JMX indicator parameters were also written to Zabbix, but there was always a lack of something to visualize an operable interface. The data in the Zabbix is more scattered, and the whole cluster cannot be concentrated. or a cluster in the broker list, write their own web-console more time-consuming, use the prototype tool to draw some management interface things, the key itself is not the front-end aspects of technology, this is relatively weak. This does not open source community provides the Kafka Web management platform Kafkaoffsetmonitor. Just get it up and running quickly. Let's not worry, let's get it straight.

Description

This app monitors the consumer of your Kafka service and their offset (offset) in partition in real time.

You can browse the current consumer group, and all partition consumption of each topic can be viewed at a glance. This is actually very useful, from here you quickly know whether each partition message is quickly consumed (without blocking). He can guide you (Kafka producer and Consumer) to optimize the code.

This web management platform retains the historical data of partition offset and consumer lag, so you can easily understand the consumer consumption of these days.

Kafkaoffsetmonitor Features:

1. From the title can be seen, Kafka offset monitor, is to monitor the consumer consumption situation, and can list each consumer offset, lag data.

2. Consumer group List

3. List of all parition for each topic (Topic,pid,offset,logsize,lag,owner)

4. View topic's historical consumption information.

Although the functional coverage is not complete, but very practical.

1. Download

GitHub website Download

Kafkaoffsetmonitor

Baidu Cloud Download (fast speed)

Baidu Cloud Kafkaoffsetmonitor Download

description : Baidu Cloud download for modified version, because kafkaoffsetmonitor some resources files (CSS,JS) is to access the outside network, especially have access to Google resources, we all understand, often can not access. Recommended Download modified version

2. Installation

Kafkaoffsetmonitor is easy to run because all the running files, resource files, jar files are packaged into the Kafkaoffsetmonitor-assembly-0.2.0.jar and run directly, this way is great. Neither compiles nor configures, hehe, nor is it absolutely not configured.

A. Create a new directory kafka-offset-console, and then copy the jar to the directory.

B. Create a new script, because you may not be a Kafka cluster. Scripts can be used to start multiple

[Email protected]: Vim mobile_start_en.sh

#!/bin/bash
JAVA-XMS512M-XMX512M-XSS1024K-XX:PERMSIZE=256M-XX:MAXPERMSIZE=512M-CP kafkaoffsetmonitor-assembly-0.2.0.jar \
COM.QUANTIFIND.KAFKA.OFFSETAPP.OFFSETGETTERWEB \
--zk 192.168.2.101:2181,192.168.2.102:2182,192.168.2.103:2181/config/mobile/xxx \
--port 8086 \
--refresh 10.seconds \
--retain 7.days 1>mobile-logs/stdout.log 2>mobile-logs/stderr.log &

Note:/config/mobile/xxx represents the root directory of ZK and needs to be created manually or without setting

3. Running

[Email protected]: chmod +x mobile_start_en.sh

[Email protected]:./mobile_start_en.sh

Serving Resources From:jar:file:/opt/xxx/kafka-offset-console/kafkaoffsetmonitor-assembly-0.2.0.jar!/offsetapp

6 Demo:

Consumer group List

Topic List of all Partiton consumption conditions

The meaning of the medium parameter is explained as follows:

Topic: Topic Name at creation time

Partition: Partition number

Offset: Indicates how many message the parition has consumed

LogSize: Indicates how many message the partition has written

Lag: Indicates how many message lines have not been consumed.

Owner: Represents the Consumer

Created: the partition creation time

Last Seen: Consumption status Refresh latest time.

Kafka running Topic

Topic list in Kafka cluster

Broker list in Kafka cluster

Apache Kafka Surveillance Series-kafkaoffsetmonitor (EXT)