Go Service Monitoring indicator (metric) escalation Open-falcon

This is a creation in Article, where the information may have evolved or changed.

1. Overview

Indicator statistics is the basis of implementation of APM (Application performance management), usually through the statistics of some indicators and reporting, we can understand the operation of the program, timely detection of problems in the program, in advance to anticipate system bottlenecks.
The current implementation of the indicator (metric) is metrics, which is a Java implementation that can be directly introduced into the program as a library. See Go-metrics for the implementation of the Go language.
In addition, this is just the indicator in memory processing and calculation, if we want to show, we need to throw the data, here can be thrown into the log, or can be thrown into the time series database, the simplest way is directly thrown into the monitoring system to draw or alarm. So this article will explain the meaning of the indicators and how to throw the calculated data into the monitoring Open-falcon

2. Indicators Statistical methods

2.1 Meters

Used to calculate a period of measurement, usually used to calculate the interface call frequency, such as QPS (number of times per second), mainly divided into ratemean,rate1/rate5/rate15 and other indicators.

• Ratemean

The number of occurrences within a unit time, such as 100 times a minute, is 100/60.

• Rate1/rate5/rate15

1 min/5 min/15 min Sliding average (moving average),

2.2 Gauges

For the measurement of instantaneous values, such as we can over a period of time on the memory of the use of statistics and reporting, then all the data point set is the corresponding time point of memory value, gauges only the value indicator. That is, what is reported is what.

2.3 Counter

Count class statistics, can be added or minus, can also be zero operation, all operations are based on the old value. This can be done by zeroing each day, and then adding a new registered user to count the daily registered users.

2.4 histograms

It is mainly used for the statistics of the distribution of the values in the data set, the typical application scenario is that the interface is time consuming, each invocation of the interface is time consuming, and it is not realistic to record each invocation time to analyze the time-consuming situation of the interface. Therefore, the time spent on the interface as a data set, and the acquisition of Count,min, Max, Mean, Median, 75%, 95%, 99% and other indicators. With a relatively small resource consumption, to reflect the real situation of the data set as much as possible.

• Count

The number of samples resulting from the last cleanup.

• Min

Minimum value in a sample

• Max

Maximum value in a sample

• Mean

Average of all samples evaluated

• Median

The value of the middle position in the sample.

• 75%

The value of the%75 position in the sample.

• 85D

The value of the%95 position in the sample.

• 99%

The value of the%99 position in the sample.

1.5 Timers

The frequency of statistical calls to a code module and the time-consuming statistics of calls. The indicator is a collection of two statistical methods, histograms and meters.

3. How to use

See Go-metric documentation for more detailed usage

3.1 Counter

c := metrics.NewCounter()metrics.Register("foo", c)//进行加操作c.Inc(47)//进行减操作c.Dec(1)//获取出值c.Count()

3.2 Gauge

g := metrics.NewGauge()metrics.Register("bar", g)//更新瞬时值g.Update(47)//获取出瞬时值g.Value()

3.3 Meters

m := metrics.NewMeter()metrics.Register("quux", m)//写入数据集m.Mark(47)//获取数据集只读快照m := metric.Snapshot()//数据集大小m.Count()//1分钟滑动平均值m.Rate1()//5分钟滑动平均值m.Rate5()//15分钟滑动平均值m.Rate15()//平均值m.RateMean()

3.4 histograms

h := metrics.NewHistogram(s)metrics.Register("baz", h)//写入数据集h.Update(47)//获取数据集只读快照h := metric.Snapshot()//数据集大小h.Count()//最小值h.Min()//最大值h.Max()//平均值h.Mean()ps := h.Percentiles([]float64{0.5, 0.75, 0.95, 0.99})//中位数ps[0]//75%的数ps[1]//95%的数ps[2]//99%的数ps[3]

3.5 Timer

t := metrics.NewTimer()metrics.Register("bang", t)t.Time(func() {    //do some thing})t.Update(47)//获取方式同meter以及Histograms

4. Report on the indicator Open-falcon

4.1 Escalation method

Code and how to use see Go-metrics-falcon

To achieve data escalation open-falcon, only need to take out all the data, according to the Open-falcon format to escalate, here are related to the definition of the report JSON, as follows.

{    "endpoint": "$endpoint",    "metric": "$name",    "value": 2.2,    "step": 60,    "counterType": "GAUGE",    "tags": "project=$projectName,metricType=meter,valueType=ratemean",    "timestamp": 1524724608}

Endpoint: This one is typically the host hostname, which is used to label which machine.
Metric: indicator name, defined by user
Value: The values of the indicator
Step: Time period of escalation
CounterType: The type of escalation, where Open-falcon only supports gauge and counter, so use gauge uniformly.
Tags: tags, for but not indicators, including indicator type, value type, project name three items.
Timestamp: The timestamp of the indicator escalation, in seconds.

4.2 Effects

, enter endpoint, and then enter the project name in the Counter section to filter out all the metrics reported by the project.

Click on the indicator to enter a larger view of the indicator.

At the same time we can monitor the indicator settings, see Open-falcon documentation.

5. Reference

Metrics
Go-metric
Apm
Open-falcon