Prometheus + Grafana monitoring deployment practices, prometheusgrafana

Prometheus + Grafana monitoring deployment practices, prometheusgrafana

Reference:

I. Environment 1. Topology

2. Nodes

Node	OS	Hostname	IP	Remark
Prometheus & grafana server	Centos 1, 7.4	Prometheus	172.20.1.211
Prometheus node	Centos 1, 7.4	Node1	172.20.1.212

3. Version (as of 20171130)

Soft/Node	Version	Download
Prometheus	2.0.0	Https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz
Node_exporter	V0.15.1	Https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz
Grafana	4.6.2-1	Https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.2-1.x86_64.rpm

 

2. Deploy prometheus

Deploy the prometheus service on the prometheus & grafana server node.

1. Download and deploy

# Download [root @ prometheus src] # cd/usr/local/src/[root @ prometheus src] # deploy the wget https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz# to the/usr/local/directory # promethus without compilation and installation, unzip the configuration file and startup file in the directory [root @ prometheus src] # tar-zxvf prometheus-2.0.0.linux-amd64.tar.gz-C/usr/local/[root @ prometheus src] # cd/usr/local /[ root @ prometheus local] # Music prometheus-2.0.0.linux-amd64/prometheus/# verify [root @ prometheus local] # cd prometheus/[root @ prometheus] #. /prometheus -- version

2. Configuration File

# Decompress prometheus in the directory. yml # simple verification. The main configuration is the default file configuration. If there is a modification/addition, the [root @ prometheus] # vim prometheus is marked in red. yml # global configuration: scrape_interval: 15 s # Set the capture (pull) interval. The default value is 1 m evaluation_interval: 15 s # set the rules evaluation interval, the default value is 1 m # scrape_timeout is set to the global default (10 s ). # alarm management configuration, not used yet. The default configuration is alerting: alertmanagers:-static_configs:-targets: #-alertmanager: 9093 # Load rules and perform regular evaluation based on the set interval, not used yet. The default configuration is rule_files :#- "First_rules.yml" #-"second_rules.yml" # capture (pull), that is, the Monitoring Target configuration # by default, only the monitoring configuration of the host is scrape_configs: # metric target label (the metric target here is only a metric, rather than a specific host. You can obtain multiple metric targets on a specific host ), this label-job_name: 'prometheus' # metrics_path ults to '/metrics' # scheme defaults to 'http' is added to each captured time series table '. # overwrite the capture interval set by the global configuration, which is rewritten from 15 seconds to 5 seconds. Scrape_interval: 5S # specifies the Monitoring Target statically, does not involve the use of some service discovery mechanisms to find the target static_configs:-targets: ['localhost: 100'] # (opentional) and then add a label, labels: instance: prometheus-job_name: 'linux 'scrape_interval: 10sstatic_configs: # use the default port-targets: ['172. listen 1.212: 9100 '] labels: instance: node1

3. Set the user

# Add a user, later, use this account to start the service [root @ prometheus prom etheus] # groupadd prometheus [root @ prometheus] # useradd-g prometheus-s/sbin/nologin prometheus # authorize [root @ promethe prometheus] # cd ~ [Root @ prometheus ~] # Chown-R prometheus: prometheus/usr/local/prometheus/# create the prometheus running data directory [root @ prometheus ~] # Mkdir-p/var/lib/prometheus [root @ prometheus ~] # Chown-R prometheus: prometheus/var/lib/prometheus/

4. Set startup

[Root @ prometheus ~] # Touch/usr/lib/systemd/system/prometheus. service [root @ prometheus ~] # Chown prometheus: prometheus/usr/lib/systemd/system/prometheus. service [root @ prometheus ~] # Vim/usr/lib/systemd/system/prometheus. service [Unit] Description = PrometheusDocumentation = https://prometheus.io/after?network.target?service=# when Type is set to running y, the service will continuously restart Type = simpleUser = prometheus # -- storage. tsdb. path is optional. The default data directory is in the running directory. execStart =/usr/local/prometheus -- config. file =/usr/local/prometheus. yml -- storage. tsdb. path =/var/lib/prometheusRestart = on-failure [Install] Wan Tedby=multi user.tar get # Set boot start [root @ prometheus ~] # Systemctl enable Prometheus [root @ prometheus ~] # Systemctl start prometheus

5. Set iptables

[root@prometheus ~]# vim /etc/sysconfig/iptables-A INPUT -p tcp -m state --state NEW -m tcp --dport 9090 -j ACCEPT[root@prometheus ~]# service iptables restart

6. Start and verify 1) view service status

[root@prometheus ~]# systemctl status prometheus

[root@prometheus ~]# netstat -tunlp | grep 9090

2) web ui

Prometheus comes with a simple UI, http: // 172.20.1.211: 9090

Under the Status menu, such as Configuration, Rule, and Targets,

Statu --> Configuration displays the Configuration of prometheus. yml, as follows:

Statu --> Targets: displays the monitoring target. Currently, node_exporter is not set for the monitoring target "linux", but scrape data is not set, as shown below:

7. Drawing

Visit: http: // 172.20.1.211: 9090/metrics to view the data captured from the exporter, as shown below:

Access: http: // 172.20.1.211: 9090. In the input box, enter one exporter to capture and obtain the value. Click the "Execute" and "Execute" buttons to view the corresponding data capture image, you can also adjust the time and unit as follows:

3. Deploy node_exporter

Node_exporter collects system data of the machine. In this example, the exporter officially provided by prometheus is used. In addition to node_exporter, exporter such as consul, memcached, haproxy, and mysqld are also provided. For details, refer to the official website.

Deploy related services on the prometheus node.

1. Download and deploy

# Download [root @ node1 ~] # Cd/usr/local/src/[root @ node1 src] # wget https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz# deployment [root @ node1 src] # tar-zxvf node_exporter-0.15.1.linux-amd64.tar.gz-C/usr/local/[root @ node1 src] # cd/usr/local/[root @ node1 local] # Music node_exporter-0.15.1.linux-amd64/node_exporter/

2. Set the user

[root@node1 ~]# groupadd prometheus[root@node1 ~]# useradd -g prometheus -s /sbin/nologin prometheus[root@node1 ~]# chown -R prometheus:prometheus /usr/local/node_exporter/

3. Set startup

[root@node1 ~]# vim /usr/lib/systemd/system/node_exporter.service[Unit]Description=node_exporterDocumentation=https://prometheus.io/After=network.target[Service]Type=simpleUser=prometheusExecStart=/usr/local/node_exporter/node_exporterRestart=on-failure[Install]WantedBy=multi-user.target[root@node1 ~]# systemctl enable node_exporter[root@node1 ~]# systemctl start node_exporter

4. Set iptables

# The official node_exporter uses port 9100 by default [root @ node1 ~] # Vim/etc/sysconfig/iptables-a input-p tcp-m state -- state NEW-m tcp -- dport 9100-j ACCEPT [root @ node1 ~] # Service iptables restart

5. Verify

Access: http: // 172.20.1.211: 9090. It can be seen that the node1 host is monitored as follows:

4. Deploying grafana

Deploy the grafana service on the prometheus & grafana server node.

1. Download and install

# Download [root @ prometheus ~] # Cd/usr/local/src/[root @ prometheus src] # wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.2-1.x86_64.rpm# installation [root @ prometheus src] # yum localinstall grafana-4.6.2-1.x86_64.rpm

2. Configuration File

The configuration file is located in/etc/grafana. ini, so keep the default configuration here.

3. Set startup

[root@prometheus src]# systemctl enable grafana-server[root@prometheus src]# systemctl start grafana-server

4. Set iptables

# Grafana-server uses port 3000 [root @ prometheus src] # vim/etc/sysconfig/iptables-a input-p tcp-m state -- state NEW-m tcp -- dport 3000 by default -j ACCEPT [root @ prometheus src] # service iptables restart

5. Add a data source 1) log on

Access: http: // 172.20.1.211: 3000, default account/password: admin/admin

2) Add a data source

On the homepage, click "Add data source" to go to the "Add data source" page. The configuration is as follows:

Name: prometheus

Type: prometheus

URL: http: // localhost: 9090/

Access: proxy

Deselect the Default check box. For other Default values, click "Add", as shown below:

On the "Dashboards" Page, sign the "import" built-in template as follows:

6. Import dashboard

Download related dashboaed from grafana official website to local, such as: https://grafana.com/dashboards/405

Grafana homepage --> icon in the upper left corner --> Dashboard --> import

Upload has been downloaded to a local json file (or use the dashboard id, such as 405), as follows:

Select "prometheus" as the data source name, and click "Import" as follows:

7. view the dashboard

Grafana homepage --> icon in the upper left corner --> Dashboard --> Home, Two Dashboards that have been added are displayed in the Home drop-down list, "Prometheus Stats" and "Node Exporter Server Metrics ", select one, as shown below: