Use ganglia to monitor storm Clusters
Recently, some tasks in the storm cluster cannot be started and executed properly due to unknown reasons (several projects are reused ), so I want to get a monitoring system to monitor the status of the storm cluster and handle the corresponding alarms. Finally, ganglia, an open-source monitoring platform, is selected. Ganglia can use jmxtrans to monitor storm. jmxtrans serves as a communication bridge between ganglia and storm.
1. Download and install jmxtrans
You can use wget to download
Wget https://cloud.github.com/downloads/jmxtrans/jmxtrans/jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm
Sudo rpm-I jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm
After installation, the jmxtrans directory is in the/usr/share/jmxtrans directory, the configuration file is in the/etc/sysconfig/jmxtrans directory, and the json script for monitoring storm is placed in the/var/lib/jmxtrans directory.
2. Modify storm Configuration
Add the following two parameters to storm. yaml:
Supervisor. childopts: "-verbose: gc-XX: + PrintGCTimeStamps-XX: + PrintGCDetails-Dcom. sun. management. jmxremote-Dcom. sun. management. jmxremote. ssl = false-Dcom. sun. management. jmxremote. authenticate = false-Dcom. sun. management. jmxremote. port = 12346"
Nimbus. childopts: "-verbose: gc-XX: + PrintGCTimeStamps-XX: + PrintGCDetails-Dcom. sun. management. jmxremote-Dcom. sun. management. jmxremote. ssl = false-Dcom. sun. management. jmxremote. authenticate = false-Dcom. sun. management. jmxremote. port = 12345"
The port numbers 12345 and 12346 can be set at will, preferably useless or infrequently used port numbers.
3. Write monitoring scripts
Supervisor. json
- {
- "servers" : [ {
- "port" : "12346",
- "host" : "IP_OF_SUPERVISOR_MACHINE",
- "queries" : [ {
- "outputWriters": [{
- "@class":"com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "supervisor",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649" }
- }],
- "obj": "java.lang:type=Memory",
- "resultAlias": "supervisor",
- "attr": ["ObjectPendingFinalizationCount"]
- },
- {
- "outputWriters": [{
- "@class":"com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "supervisor",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649"
- }
- }],
- "obj": "java.lang:name=Copy,type=GarbageCollector",
- "resultAlias": "supervisor",
- "attr": [
- "CollectionCount",
- "CollectionTime"
- ]
- },
- {
- "outputWriters": [{
- "@class": "com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "supervisor",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649"
- }
- }],
- "obj": "java.lang:name=Code Cache,type=MemoryPool",
- "resultAlias": "supervisor",
- "attr": [
- "CollectionUsageThreshold",
- "CollectionUsageThresholdCount",
- "UsageThreshold",
- "UsageThresholdCount"
- ]
- },
- {
- "outputWriters": [{
- "@class": "com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "supervisor",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649"
- }
- }],
- "obj": "java.lang:type=Runtime",
- "resultAlias": "supervisor",
- "attr": [
- "StartTime",
- "Uptime"
- ]
- }],
- "numQueryThreads" : 2
- }]
- }
Port 12346 is the port of the jmx service, which is set in storm. yaml;
IP_OF_SUPERVISOR_MACHINE is the IP address of the supervisor node.
IP_OF_GANGLIA_GMOND_SERVER is the IP address of the server where the ganglia gmond service is located.
Nimbus. json
- {
- "servers" : [{
- "port" : "12345",
- "host" : "IP_OF_NIMBUS_MACHINE",
- "queries" : [
- { "outputWriters": [{
- "@class": "com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "nimbus",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649"
- }
- }],
- "obj": "java.lang:type=Memory",
- "resultAlias": "nimbus",
- "attr": ["ObjectPendingFinalizationCount"]
- },
- {
- "outputWriters": [{
- "@class": "com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "nimbus",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649"
- }
- }],
- "obj": "java.lang:name=Copy,type=GarbageCollector",
- "resultAlias": "nimbus",
- "attr": [
- "CollectionCount",
- "CollectionTime"
- ]
- },
- {
- "outputWriters": [{
- "@class": "com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "nimbus",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649"
- }
- }],
- "obj": "java.lang:name=Code Cache,type=MemoryPool",
- "resultAlias": "nimbus",
- "attr": [
- "CollectionUsageThreshold",
- "CollectionUsageThresholdCount",
- "UsageThreshold",
- "UsageThresholdCount"
- ]
- },
- {
- "outputWriters": [{
- "@class": "com.googlecode.jmxtrans.model.output.GangliaWriter",
- "settings": {
- "groupName": "nimbus",
- "host": "IP_OF_GANGLIA_GMOND_SERVER",
- "port": "8649"
- }
- }],
- "obj": "java.lang:type=Runtime",
- "resultAlias": "nimbus",
- "attr": [
- "StartTime",
- "Uptime"
- ]
- }],
- "numQueryThreads" : 2
- } ]
- }
Port 12345 is the port of the jmx service, which is set in storm. yaml;
IP_OF_NIMBUS_MACHINE is the IP address of the nimbus node.
IP_OF_GANGLIA_GMOND_SERVER is the IP address of the server where the ganglia gmond service is located.
4. Copy the nimbus. json and supervisor. json scripts to the/var/lib/jmxtrans directory.
5. Start the jmxtrans Service
Cd/usr/share/jmxtrans
Sudo./jmxtrans strart
6. Now you can view the running status of the storm cluster on the wen service of ganglia.