Ganglia monitoring custom metric practices, gangliametric
The Ganglia monitoring system is an open-source UC Berkeley project designed to monitor distributed clusters. The monitoring level includes the resource level and business level, the resource layer includes cpu, memory, disk, IO, and network load. As for the business layer, you can easily add custom metric, therefore, it can be used for monitoring such as service performance, load, and error rate, such as the QPS and Http status error rate of a web service. In addition, if it is integrated with Nagios, an alarm can be triggered when an indicator exceeds a certain threshold.
Compared with zabbix, Ganglia has a low system overhead caused by the client collection agent (gmond) and does not affect the performance of related services.
Ganglia mainly has several modules:
- Gmond: deployed on various monitored machines for periodically collecting data for broadcast or unicast.
- Gmetad: deployed on the server, regularly pulls the data collected by gmond from the host in the configured data_source.
- Ganglia-web: ship monitoring data to the web page
About the installation of ganglia this article does not introduce too much, the portal: http://www.it165.net/admin/html/201302/770.html
This document describes how to develop custom metric to monitor metrics that you are concerned about.
There are several main methods:
1. Use gmetric directly
The machine where gmond is installed will install/usr/bin/gmetric at the same time. This command is a tool that broadcasts information such as a metric name value, for example
/usr/bin/gmetric -c /etc/ganglia/gmond.conf --name=test --type=int32 --units=sec --value=2
For more information about gmetric options, see: http://manpages.ubuntu.com/manpages/hardy/man1/gmetric.1.html
In addition to using gmetric through command lines, you can also use binding in common languages, such as go, Java, and python. Related binding can be used on github, you only need to import it in. Go language https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-go
Ruby https://github.com/igrigorik/gmetric/blob/master/lib/gmetric.rb
Java https://github.com/ganglia/ganglia_contrib/tree/master/gmetric-java
Python https://github.com/ganglia/ganglia_contrib/tree/master/gmetric-python
2. Use a third-party tool based on gmetric
This paper uses ganglia-logtailer as an example: https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-logtailer
This tool is based on the logtail (debain)/logcheck (centos) package to implement regular tail of logs, and then uses the corresponding class to analyze logs by specifying classname,
The custom metric is calculated based on the fields that interest you, and is broadcast by gmetric.
For example, you can modify NginxLogtailer. py according to the nginx log format of your service as follows:
# -*- coding: utf-8 -*-###### This plugin for logtailer will crunch nginx logs and produce these metrics:### * hits per second### * GETs per second### * average query processing time### * ninetieth percentile query processing time### * number of HTTP 200, 300, 400, and 500 responses per second###### Note that this plugin depends on a certain nginx log format, documented in## __init__.import timeimport threadingimport re# local dependenciesfrom ganglia_logtailer_helper import GangliaMetricObjectfrom ganglia_logtailer_helper import LogtailerParsingException, LogtailerStateExceptionclass NginxLogtailer(object): # only used in daemon mode period = 30 def __init__(self): '''This function should initialize any data structures or variables needed for the internal state of the line parser.''' self.reset_state() self.lock = threading.RLock() # this is what will match the nginx lines #log_format ganglia-logtailer # '$host ' # '$server_addr ' # '$remote_addr ' # '- ' # '"$time_iso8601" ' # '$status ' # '$body_bytes_sent ' # '$request_time ' # '"$http_referer" ' # '"$request" ' # '"$http_user_agent" ' # '$pid'; # NOTE: nginx 0.7 doesn't support $time_iso8601, use $time_local # instead # original apache log format string: # %v %A %a %u %{%Y-%m-%dT%H:%M:%S}t %c %s %>s %B %D \"%{Referer}i\" \"%r\" \"%{User-Agent}i\" %P # host.com 127.0.0.1 127.0.0.1 - "2008-05-08T07:34:44" - 200 200 371 103918 - "-" "GET /path HTTP/1.0" "-" 23794 # match keys: server_name, local_ip, remote_ip, date, status, size, # req_time, referrer, request, user_agent, pid self.reg = re.compile('^(?P<remote_ip>[^ ]+) (?P<server_name>[^ ]+) (?P
After ganglia-logtailer is deployed on the monitored machine, run the following command to create a crond task:
*/1 * root/usr/local/bin/ganglia-logtailer -- classname NginxLogtailer -- log_file/usr/local/nginx-video/logs/access. log -- mode cron -- gmetric_options '-C test_cluster-g nginx_status'
Reload crond service. After one minute, you can see the corresponding metric Information On ganglia web:
For the deployment method of ganglia-logtailer, see: https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-logtailer
3. Write your own module in Supported languages. This article uses Python as an example.
Ganglia allows you to write your own Python module. The following is a brief introduction to github:
Writing a Python module is very simple. You just need to write it following a template and put the resulting Python module (. py) in/usr/lib (64)/ganglia/python_modules.
A corresponding Python Configuration (. pyconf) file needs to reside in/etc/ganglia/conf. d /.
For example, compile an example Python file to check machine Temperature
acpi_file = "/proc/acpi/thermal_zone/THRM/temperature"def temp_handler(name): try: f = open(acpi_file, 'r') except IOError: return 0 for l in f: line = l.split() return int(line[1])def metric_init(params): global descriptors, acpi_file if 'acpi_file' in params: acpi_file = params['acpi_file'] d1 = {'name': 'temp', 'call_back': temp_handler, 'time_max': 90, 'value_type': 'uint', 'units': 'C', 'slope': 'both', 'format': '%u', 'description': 'Temperature of host', 'groups': 'health'} descriptors = [d1] return descriptorsdef metric_cleanup(): '''Clean up the metric module.''' pass#This code is for debugging and unit testingif __name__ == '__main__': metric_init({}) for d in descriptors: v = d['call_back'](d['name']) print 'value for %s is %u' % (d['name'], v)
With the module function file, you also need to write a corresponding configuration file (under/etc/ganglia/conf. d/temp. pyconf) in the following format:
modules { module { name = "temp" language = "python" # The following params are examples only # They are not actually used by the temp module param RandomMax { value = 600 } param ConstantValue { value = 112 } }}collection_group { collect_every = 10 time_threshold = 50 metric { name = "temp" title = "Temperature" value_threshold = 70 }}
With these two files, this module is successfully added.
For more modules contributed by users, see https://github.com/ganglia/gmond_python_modules
This module is useful for monitoring metric of common services such as elasticsearch, filecheck, nginx_status, and MySQL. You only need to make some modifications to meet your needs.
Other useful tools for user contribution
- Ganglia-alert: Get gmetad data and alert https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-alert
- Ganglia-docker: Use ganglia, https://github.com/ganglia/ganglia_contrib/tree/master/docker in docker
- Gmetad-health-check: monitoring the gmetad service status, restart service, https://github.com/ganglia/ganglia_contrib/tree/master/gmetad_health_checker if down
- Chef-ganglia: deploy ganglia with chef, https://github.com/ganglia/chef-ganglia
- Ansible-ganglia: automated deployment of ganglia with ansible, https://github.com/remysaissy/ansible-ganglia
- Ganglia-nagios: integrated with nagios and ganglia, https://github.com/ganglia/ganglios
- Ganglia-api: provide external rest api, return the data collected by gmetad in a specific format, https://github.com/guardian/ganglia-api
If you have any questions, please leave a message.
Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.