Use JMX to monitor Kafka

Use JMX to monitor Kafka

Kafka can be configured to use JMX to monitor the running status. You can use JDK's built-in Jconsole to observe the results or use Java APIs.

For the description of monitoring indicators, you can refer to: http://kafka.apache.org/documentation.html#monitoring

Enable JMX Port

Modify the bin/kafka-server-start.sh and add the JMX_PORT parameter as follows

If ["x $ KAFKA_HEAP_OPTS" = "x"]; then
Export KAFKA_HEAP_OPTS = "-Xmx1G-Xms1G"
Export maid = "9999"
Fi

You can connect to the Jconsole during testing.

Access through Java API

Obtain the target value using the following methods:

Public class KafkaDataProvider {
Protected final Logger LOGGER = LoggerFactory. getLogger (getClass ());
Private static final String MESSAGE_IN_PER_SEC = "kafka. server: type = BrokerTopicMetrics, name = MessagesInPerSec ";
Private static final String BYTES_IN_PER_SEC = "kafka. server: type = BrokerTopicMetrics, name = BytesInPerSec ";
Private static final String BYTES_OUT_PER_SEC = "kafka. server: type = BrokerTopicMetrics, name = BytesOutPerSec ";
Private static final String PRODUCE_REQUEST_PER_SEC = "kafka. network: type = RequestMetrics, name = RequestsPerSec, request = Produce ";
Private static final String CONSUMER_REQUEST_PER_SEC = "kafka. network: type = RequestMetrics, name = RequestsPerSec, request = FetchConsumer ";
Private static final String FLOWER_REQUEST_PER_SEC = "kafka. network: type = RequestMetrics, name = RequestsPerSec, request = FetchFollower ";
Private static final String ACTIVE_CONTROLLER_COUNT = "kafka. controller: type = KafkaController, name = ActiveControllerCount ";
Private static final String PART_COUNT = "kafka. server: type = ReplicaManager, name = PartitionCount ";
Public String extractMonitorData (){
// TODO obtains the IP address and parameters by calling the API.
KafkaRoleInfo monitorDataPoint = new KafkaRoleInfo ();
String jmxURL = "service: jmx: rmi: // jndi/rmi: // 192.168.40.242: 9999/jmxrmi ";
Try {
MBeanServerConnection jmxConnection = MetricDataUtils. getMBeanServerConnection (jmxURL );
ObjectName messageCountObj = new ObjectName (MESSAGE_IN_PER_SEC );
ObjectName bytesInPerSecObj = new ObjectName (BYTES_IN_PER_SEC );
ObjectName bytesOutPerSecObj = new ObjectName (BYTES_OUT_PER_SEC );
ObjectName produceRequestsPerSecObj = new ObjectName (PRODUCE_REQUEST_PER_SEC );
ObjectName consumerRequestsPerSecObj = new ObjectName (CONSUMER_REQUEST_PER_SEC );
ObjectName flowerRequestsPerSecObj = new ObjectName (FLOWER_REQUEST_PER_SEC );
ObjectName activeControllerCountObj = new ObjectName (ACTIVE_CONTROLLER_COUNT );
ObjectName partCountObj = new ObjectName (PART_COUNT );
Long messagesInPerSec = (Long) jmxConnection. getAttribute (messageCountObj, "Count ");
Long bytesInPerSec = (Long) jmxConnection. getAttribute (bytesInPerSecObj, "Count ");
Long bytesOutPerSec = (Long) jmxConnection. getAttribute (bytesOutPerSecObj, "Count ");
Long produceRequestCountPerSec = (Long) jmxConnection. getAttribute (produceRequestsPerSecObj, "Count ");
Long consumerRequestCountPerSec = (Long) jmxConnection. getAttribute (consumerRequestsPerSecObj, "Count ");
Long flowerRequestCountPerSec = (Long) jmxConnection. getAttribute (flowerRequestsPerSecObj, "Count ");
Integer activeControllerCount = (Integer) jmxConnection. getAttribute (activeControllerCountObj, "Value ");
Integer partCount = (Integer) jmxConnection. getAttribute (partCountObj, "Value ");
MonitorDataPoint. setMessagesInPerSec (messagesInPerSec );
MonitorDataPoint. setBytesInPerSec (bytesInPerSec );
MonitorDataPoint. setBytesOutPerSec (bytesOutPerSec );
MonitorDataPoint. setProduceRequestCountPerSec (produceRequestCountPerSec );
MonitorDataPoint. setConsumerRequestCountPerSec (consumerRequestCountPerSec );
MonitorDataPoint. setFlowerRequestCountPerSec (flowerRequestCountPerSec );
MonitorDataPoint. setActiveControllerCount (activeControllerCount );
MonitorDataPoint. setPartCount (partCount );
} Catch (IOException e ){
E. printStackTrace ();
} Catch (MalformedObjectNameException e ){
E. printStackTrace ();
} Catch (AttributeNotFoundException e ){
E. printStackTrace ();
} Catch (MBeanException e ){
E. printStackTrace ();
} Catch (ReflectionException e ){
E. printStackTrace ();
} Catch (InstanceNotFoundException e ){
E. printStackTrace ();
}
Return monitorDataPoint. toString ();
}
Public static void main (String [] args ){
System. out. println (new KafkaDataProvider (). extractMonitorData ());
}
/**
* Obtain the MBeanServer connection
*
* @ Param jmxUrl
* @ Return
* @ Throws IOException
*/
Public MBeanServerConnection getMBeanServerConnection (String jmxUrl) throws IOException {
JMXServiceURL url = new JMXServiceURL (jmxUrl );
Jmxconnejmxc = JMXConnectorFactory. connect (url, null );
MBeanServerConnection mbsc = jmxc. getMBeanServerConnection ();
Return mbsc;
}
}

Other tools
In addition to writing customized monitoring programs

Kafka-web-console
Https://github.com/claudemamo/kafka-web-console
Deploy sbt:
Http://www.scala-sbt.org/0.13/tutorial/Manual-Installation.html
Http://www.scala-sbt.org/release/tutorial/zh-cn/Installing-sbt-on-Linux.html

KafkaOffsetMonitor
Https://github.com/quantifind/KafkaOffsetMonitor/releases/tag/v0.2.0
Java-cp KafkaOffsetMonitor-assembly-0.2.0.jar com. quantifind. kafka. offsetapp. OffsetGetterWeb -- zk localhost: 12181 -- port 8080 -- refresh 5. minutes -- retain 1.day

Mx4jLoader

Kafka architecture design of the distributed publish/subscribe message system

Apache Kafka code example

Apache Kafka tutorial notes

Principles and features of Apache kafka (0.8 V)

Kafka deployment and code instance

Introduction to Kafka and establishment of Cluster Environment

For details about Kafka, click here
Kafka: click here

This article permanently updates the link address: