Ganglia monitoring custom metric practices, gangliametric

Ganglia monitoring custom metric practices, gangliametric

The Ganglia monitoring system is an open-source UC Berkeley project designed to monitor distributed clusters. The monitoring level includes the resource level and business level, the resource layer includes cpu, memory, disk, IO, and network load. As for the business layer, you can easily add custom metric, therefore, it can be used for monitoring such as service performance, load, and error rate, such as the QPS and Http status error rate of a web service. In addition, if it is integrated with Nagios, an alarm can be triggered when an indicator exceeds a certain threshold.

Compared with zabbix, Ganglia has a low system overhead caused by the client collection agent (gmond) and does not affect the performance of related services.

Ganglia mainly has several modules:

• Gmond: deployed on various monitored machines for periodically collecting data for broadcast or unicast.
• Gmetad: deployed on the server, regularly pulls the data collected by gmond from the host in the configured data_source.
• Ganglia-web: ship monitoring data to the web page

About the installation of ganglia this article does not introduce too much, the portal: http://www.it165.net/admin/html/201302/770.html

This document describes how to develop custom metric to monitor metrics that you are concerned about.

There are several main methods:

1. Use gmetric directly

The machine where gmond is installed will install/usr/bin/gmetric at the same time. This command is a tool that broadcasts information such as a metric name value, for example

/usr/bin/gmetric -c /etc/ganglia/gmond.conf --name=test --type=int32 --units=sec --value=2    

For more information about gmetric options, see: http://manpages.ubuntu.com/manpages/hardy/man1/gmetric.1.html

In addition to using gmetric through command lines, you can also use binding in common languages, such as go, Java, and python. Related binding can be used on github, you only need to import it in. Go language https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-go
Ruby https://github.com/igrigorik/gmetric/blob/master/lib/gmetric.rb
Java https://github.com/ganglia/ganglia_contrib/tree/master/gmetric-java
Python https://github.com/ganglia/ganglia_contrib/tree/master/gmetric-python

2. Use a third-party tool based on gmetric

This paper uses ganglia-logtailer as an example: https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-logtailer

This tool is based on the logtail (debain)/logcheck (centos) package to implement regular tail of logs, and then uses the corresponding class to analyze logs by specifying classname,

The custom metric is calculated based on the fields that interest you, and is broadcast by gmetric.

For example, you can modify NginxLogtailer. py according to the nginx log format of your service as follows:

# -*- coding: utf-8 -*-######  This plugin for logtailer will crunch nginx logs and produce these metrics:###    * hits per second###    * GETs per second###    * average query processing time###    * ninetieth percentile query processing time###    * number of HTTP 200, 300, 400, and 500 responses per second######  Note that this plugin depends on a certain nginx log format, documented in##   __init__.import timeimport threadingimport re# local dependenciesfrom ganglia_logtailer_helper import GangliaMetricObjectfrom ganglia_logtailer_helper import LogtailerParsingException, LogtailerStateExceptionclass NginxLogtailer(object):    # only used in daemon mode    period = 30    def __init__(self):        '''This function should initialize any data structures or variables        needed for the internal state of the line parser.'''        self.reset_state()        self.lock = threading.RLock()        # this is what will match the nginx lines        #log_format ganglia-logtailer        #    '$host '        #    '$server_addr '        #    '$remote_addr '        #    '- '        #    '"$time_iso8601" '        #    '$status '        #    '$body_bytes_sent '        #    '$request_time '        #    '"$http_referer" '        #    '"$request" '        #    '"$http_user_agent" '        #    '$pid';        # NOTE: nginx 0.7 doesn't support $time_iso8601, use $time_local        # instead        # original apache log format string:        # %v %A %a %u %{%Y-%m-%dT%H:%M:%S}t %c %s %>s %B %D \"%{Referer}i\" \"%r\" \"%{User-Agent}i\" %P        # host.com 127.0.0.1 127.0.0.1 - "2008-05-08T07:34:44" - 200 200 371 103918 - "-" "GET /path HTTP/1.0" "-" 23794        # match keys: server_name, local_ip, remote_ip, date, status, size,        #               req_time, referrer, request, user_agent, pid        self.reg = re.compile('^(?P<remote_ip>[^ ]+) (?P<server_name>[^ ]+) (?P



After ganglia-logtailer is deployed on the monitored machine, run the following command to create a crond task:
*/1 * root/usr/local/bin/ganglia-logtailer -- classname NginxLogtailer -- log_file/usr/local/nginx-video/logs/access. log -- mode cron -- gmetric_options '-C test_cluster-g nginx_status'
Reload crond service. After one minute, you can see the corresponding metric Information On ganglia web:


For the deployment method of ganglia-logtailer, see: https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-logtailer


3. Write your own module in Supported languages. This article uses Python as an example.
Ganglia allows you to write your own Python module. The following is a brief introduction to github:
Writing a Python module is very simple. You just need to write it following a template and put the resulting Python module (. py) in/usr/lib (64)/ganglia/python_modules.
A corresponding Python Configuration (. pyconf) file needs to reside in/etc/ganglia/conf. d /.
For example, compile an example Python file to check machine Temperature


acpi_file = "/proc/acpi/thermal_zone/THRM/temperature"def temp_handler(name):      try:        f = open(acpi_file, 'r')    except IOError:        return 0    for l in f:        line = l.split()    return int(line[1])def metric_init(params):    global descriptors, acpi_file    if 'acpi_file' in params:        acpi_file = params['acpi_file']    d1 = {'name': 'temp',        'call_back': temp_handler,        'time_max': 90,        'value_type': 'uint',        'units': 'C',        'slope': 'both',        'format': '%u',        'description': 'Temperature of host',        'groups': 'health'}    descriptors = [d1]    return descriptorsdef metric_cleanup():    '''Clean up the metric module.'''    pass#This code is for debugging and unit testingif __name__ == '__main__':    metric_init({})    for d in descriptors:        v = d['call_back'](d['name'])        print 'value for %s is %u' % (d['name'],  v)

With the module function file, you also need to write a corresponding configuration file (under/etc/ganglia/conf. d/temp. pyconf) in the following format:


modules {  module {    name = "temp"    language = "python"    # The following params are examples only    #  They are not actually used by the temp module    param RandomMax {      value = 600    }    param ConstantValue {      value = 112    }  }}collection_group {  collect_every = 10  time_threshold = 50  metric {    name = "temp"    title = "Temperature"    value_threshold = 70  }}



With these two files, this module is successfully added.
For more modules contributed by users, see https://github.com/ganglia/gmond_python_modules
This module is useful for monitoring metric of common services such as elasticsearch, filecheck, nginx_status, and MySQL. You only need to make some modifications to meet your needs.

Other useful tools for user contribution

 

 
  
Ganglia-alert: Get gmetad data and alert https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-alert
 
  
Ganglia-docker: Use ganglia, https://github.com/ganglia/ganglia_contrib/tree/master/docker in docker
 
  
Gmetad-health-check: monitoring the gmetad service status, restart service, https://github.com/ganglia/ganglia_contrib/tree/master/gmetad_health_checker if down
 
  
Chef-ganglia: deploy ganglia with chef, https://github.com/ganglia/chef-ganglia
 
  
Ansible-ganglia: automated deployment of ganglia with ansible, https://github.com/remysaissy/ansible-ganglia
 
  
Ganglia-nagios: integrated with nagios and ganglia, https://github.com/ganglia/ganglios
 
  
Ganglia-api: provide external rest api, return the data collected by gmetad in a specific format, https://github.com/guardian/ganglia-api

 


If you have any questions, please leave a message.

 
Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.