Prometheus and Alertmanager Combat configuration

Prometheus Timing Database

First, PROMETHEUS1, Prometheus Installation

1) Source Code installation

Prometheus Install package latest version: prometheus.io/download/

wget https://github.com/prometheus/prometheus/releases/download/v2.3.2/ prometheus-2.3.2.linux-amd64.tar.gztar txvf prometheus-2.3. 2. Linux-amd64. Tar . GZCD Prometheus-2.3. 2. Linux-amd64. Tar . gz. /prometheus--config. file=prometheus.yml

Note: by executing

./prometheus-h

You can view the specific execution parameters, which can be viewed by default after the parameters. As shown in.

2) Docker mode installation (if Docker is installed)

Creating a directory and Prometheus configuration file

mkdir //prometheus/prometheus.yml

Note: The configuration for the Prometheus.yml file is described in more detail later.

Pull Prometheus Image

Docker Pull Prom/prometheus

Start Prometheus

9090:9090 --name prometheus-v/home/prometheus.yml:/etc/prometheus/prometheus.yml prom/ Prometheus

Note: A brief description of the parameters

The A,-D option launches the Prometheus container in standalone mode, which means that the container will start in the background, in which case only stop Docker can turn off Prometheus and cannot execute CTRL + C

B,-p Select the specified port number mapping, access to the 9090 port of the machine, you can access the port of the Prometheus container 9090

C 、--name to specify container names

D,-v option to create a mapping of native files and files within Docker

E 、--config.file Specify a configuration file to run Prometheus within Docker

2, Prometheus configuration file Settings

The Prometheus configuration file is a Yaml file, and the Yaml file is written in the following requirements:

Case sensitive use indentation to indicate that a hierarchical relationship is indented without using the TAB key, only spaces are allowed. The number of spaces you indent is not important, as long as the same level elements are left aligned

Examples of PROMETHEUS.YML

# Prometheus Global Configuration Entry  Scrape_interval:     15s # Sets the period for fetching data, which defaults to 1min  evaluation_interval:15s # Set the period of update rules file, default is 1min  scrape_timeout:15s # Set the time-out for fetching data, default is 10s
  External_labels: # Additional attributes will be added to the pull to get the data to coexist in the database
   Monitor: ' Codelab_monitor '
# Alertmanager Configuration alerting:alertmanagers:-static_configs:   -Targets: ["localhost:9093"] # Sets the interface for Alertmanager and Prometheus Interactions, that is, the IP address and port # rule configuration for Alertmanager listening,     first reading the default load, and then loading the rule according to the period set by the Evaluation_interval _files:-"alertmanager_rules.yml"-"prometheus_rules.yml" # Scape configuration scrape_configs:-job_name: ' Prometheus ' # job_ Name is written by default in TimeSeries labels, which can be used for queries using the  scrape_interval:15s # crawl cycle, by default with global configuration  static_configs: # static Configuration  - Targets: [' localdns:9090 '] # Prometheus the address of the data to fetch, i.e. the instance instance item-job_name: ' Example-random '  static_configs:  -Targets: [' localhost:8080 ']

3, dynamic update Prometheus configuration items

Dynamic Update Prometheus Configuration, that is, hot update load, a total of two ways:

1) Send Sighup signal to Prometheus Process

2) curl-x POST http://localdns:9090/-/reload

Reference Link: songjiayang.gitbooks.io/prometheus/content/qa/hotreload.html

4. Prometheus Data Display

Here are two ways to visualize the Prometheus data.

1) Expression Browser

In the browser, enter the machine IP address and port number to deploy the Prometheus database

Http://localdns:9090/graph

The interface is displayed as follows, so you can view the data in the Prometheus via a browser.

2) Grafana graphical interface

Installation start

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/ grafana-5.2.3.linux-amd64.tar.gztar zxvf grafana-5.2. 3. Linux-amd64. Tar . GZCD Grafana-5.2. 3. Linux-amd64. Tar . Gzbin/grafana-server Web

Grafana The default service port number is 3000, which is accessed through the browser to the Grafana.

http://localdns:3000

The default login password is "Admin/admin".

Create a Prometheus data source:

1 In the left-hand toolbar, click the Configuration menu.  2 click on "Data Sources".  3 Click "Add data Source".  4 data Source Type Select "Prometheus".  5 set the Prometheus Service access address (for example: http://localhost:9090).  6 Adjust the other desired settings (for example: Turn off delegate access).  7 Click the "Add" button to save this new data source. 

After that, the data is displayed by adding a dashboard (dashboards).

Second, Alertmanager (email alarm) 1, Alertmanager installation

SOURCE Installation

mkdir -P $GOPATH/src/github.com/prometheuscd $GOPATH/src/github.com/prometheusgit clone HTTPS ://github.com/prometheus/alertmanager.gitcd alertmanager make Build

Start

./alertmanager-config. file= alertmanager.yml #默认配置项为alertmanager. yml

Note: The ALERTMANAGER.YML configuration file, which does not exist by default, needs to be created.

2, the ALERTMANAGER.YML configuration

# Global Configuration Entry global:resolve_timeout:5m #处理超时时间, default is 5min smtp_smarthost: ' smtp.sina.com:25 ' # Mailbox SMTP Server Proxy smtp_from: ' ****** @sina. com ' # Send mailbox name smtp_auth_username: ' ****** @sina. com ' # mailbox name Smtp_auth_password: ' ****** ' # mailbox password or authorization code
Wechat_api_url: ' qyapi.weixin.qq.com/cgi-bin/' # corporate address

# define Template Confidence
Templates
-' Template/*.tmpl '
# define the routing tree Information route:group_by: [' Alertname '] # Alarm GROUP BY Group_wait:10s # Initially the first time you wait for a notification to send a set of alerts Group_interval:10s # before sending a new alert Waiting time Repeat_interval:1m # The period of sending duplicate alerts for the email configuration, this entry can not be set too low, otherwise it will be sent too much mail, the SMTP server rejected receiver: ' email ' # The name of the recipient sending the alert, the following R Eceivers Name # defines the alert receiver information receivers:-Name: ' Email ' # Alert email_configs: # Mailbox configuration-to: ' ******@163.com ' # receive an alert EMA Il configuration
HTML: ' {{template ' test.html '.}} ' # Set the content template for the mailbox
Headers: {Subject: "[WARN] Alarm Message"} # receive the title of the message
Webhook_configs: # Webhook Configuration
-url: ' http://127.0.0.1:5001 '
Send_resolved:true
Wechat_configs: # Enterprise Alarm Configuration
-Send_resolved:true
To_party: ' 1 ' # Receive group ID
agent_id: ' 1000002 ' # (Enterprise-to-custom application-->agentid)
corp_id: ' ****** ' # Corporate information (my business-->corpid[at the bottom])
Api_secret: ' ****** ' # Enterprise(Enterprise-to-custom application-->secret)
Message: ' {{template ' test_wechat.html '.}} ' # Send message template settings

  
     
    
     
    Equal: [' alertname ', ' dev ', ' instance ']

Note:

1) repeat_interval configuration items, for email, this item can not be set too low, otherwise it will be sent too much mail, by the SMTP server rejected

2) Business registration address: work.weixin.qq.com

The above configuration of email, Webhook and WeChat three kinds of alarm methods. At present Alertmanager all the alarm way has the following aspects:

Email_confighipchat_configpagerduty_configpushover_configslack_configopsgenie_configvictorops_config

3. Configuration of the. Tmpl template

1) Test.tmpl

{{define ' test.html '}}<TableBorder= "1">        <TR>                <TD>Alarm Item</TD>                <TD>Instance</TD>                <TD>Alarm Valve Value</TD>                <TD>Start time</TD>        </TR>{{range $i, $alert: =. Alerts}}<TR>                        <TD>{{index $alert.} Labels "Alertname"}}</TD>                        <TD>{{index $alert.} Labels "Instance"}}</TD>                        <TD>{{index $alert.} Annotations "Value"}}</TD>                        <TD>{{$alert.} Startsat}}</TD>                </TR>{{end}}</Table>{{end}}

Note: The above labels item represents an optional label item inside the Prometheus. The annotation entry represents the contents of the annotation item defined in the alert rule.

2) Test_wechat.tmpl

{{define ' cdn_live_wechat.html '}}  {{range $i, $alert: =. Alerts.firing}}    [Alarm entry]:{{index $alert. Labels "Alertname"}}    [Instance]:{{index $alert. Labels "Instance"}}    [alarm threshold Value]:{{index $alert. Annotations "Value"}}    [Start time]:{{$alert. Startsat}}  {{End}}{{end}}

Note: Here the range traversal item is slightly different from the email template, traversing only alarms that are not currently processed (firing). If this item is not set, an alarm item that has been resolved in alert will also be sent to the enterprise.

4. Define the alarm rules in the Prometheus module

Alertmanager_rules.yml Sample configuration file (same directory as Prometheus)

Groups:-Name:test-rules   Rules:   -Alert:instancedown # alarm name     Expr:up = = 0 # Alarm Determination conditions, refer to Prometheus advanced query to set 
   FOR:2M # When the alarm condition is satisfied how long does it take to send     an alarm labels: #标签项      team:node     Annotations: # Parse the item and explain the alarm information in detail      summary: "{{ $labels. Instance}}: Have been down "      Description:" {{$labels. Instance}}: Job {$labels. Job}} have been down "
      Value: {{$value}}

5. The 3 state of the alarm information life cycle

1) Inactive: Indicates that the current alarm information is not firing status or pending state

2) Pending: Indicates that it is activated within the set threshold time range

3) Firing: Indicates that the threshold time over the setting is activated

Iii. Display of results

Start Prometheus and Alertmanager, meet the alarm conditions, you can receive the alarm mail.

1, browser interface alarm display

In the browser input alertmanager configuration address, you can view the monitored alarm information

Http://localdns:9093/#/alerts

：

2, Mailbox alarm display

1) original Mailbox alarm Display

2) template Mailbox Alarm Display

3, Enterprise alarm display

Reference Links:

www.kancloud.cn/cdh0805010118/prometheus/719379

Songjiayang.gitbooks.io/prometheus/content/alertmanager/what.html

www.kancloud.cn/cdh0805010118/prometheus/719380

76008594?locationnum=10&fps=1

blog.qikqiak.com/post/alertmanager-of-prometheus-in-practice/