Kubernetes Cluster Monitoring Scheme

This article describes the use of Node-exporter, Prometheus, Grafana to monitor the cluster in a k8s cluster.
Its implementation principle is somewhat similar to elk, EFK combination. The Node-exporter component collects the metrics monitoring data on the node and pushes the data to Prometheus, Prometheus is responsible for storing the data, and grafana the data to the user in a graphical form on the Web page.

Before you begin, it is necessary to understand what the next Prometheus is.
Prometheus (Chinese name: Prometheus) is an open source surveillance alarm system and Time series database (TSDB) developed by SoundCloud. Since 2012, many companies and organizations have adopted Prometheus, And the project has a very active developer and user community. It has become an independent open source project. Prometheus joined CNCF (Cloud Native Computing Foundation) in 2016 as the second project to be chaired by the Foundation after Kubernetes. Prometheus's implementation is referenced by Google's internal monitoring implementation, and is well-suited to kubernetes from Google. In addition, compared to the INFLUXDB scheme, the performance is more prominent, but also built-in alarm function. It is designed for the large-scale cluster environment to pull the data collection method, only need to implement a metrics interface in the application, and then this interface to tell Prometheus can complete the data collection, for the Prometheus architecture diagram.

Prometheus Features:
1, multidimensional data model (timing column data consists of metric name and a set of Key/value)
2, flexible query language in multidimensional (PROMQL)
3, independent of distributed storage, single master node work .
4. Acquisition of time series data via an HTTP pull method
5, sequential column data push via an intermediary gateway (pushing)
6, target server can be implemented via discovery service or static configuration
7, multiple visualizations and dashboard support

Prometheus related components, the Prometheus ecosystem consists of multiple components, many of which are optional:
1. Prometheus Master Service for capturing and storing timing data
2. The client library is used to construct the application or exporter code (GO,JAVA,PYTHON,RUBY)
3. Push gateway can be used to support short connection tasks
4, visualization of the dashboard (two options, Promdash and Grafana. Current mainstream selection is Grafana.)
4, some special needs of data export (for Haproxy, STATSD, graphite and other services)
5, the experimental Alarm management end (Alartmanager, separate alarm summary, distribution, shielding, etc.)

The components of promethues are basically written in Golang, and are very friendly to compile and deploy. And there is no special dependency. Basically, they work independently.
The above text comes from the network!

Now we are officially starting the deployment process.
I. Introduction to the Environment
Operating system environment: CentOS Linux 7.2 64bit
k8s software version: 1.9.0 (deployed in Kubeadm mode)
Master Node ip:192.168.115.5/24
Node Nodes ip:192.168.115.6/24

Second, download the desired image on all nodes of the k8s cluster

# docker pull prom/node-exporter# docker pull prom/prometheus:v2.0.0# docker pull grafana/grafana:4.2.0

Iii. deployment of Node-exporter components using Daemonset method

# cat node-exporter.yaml ---apiVersion: extensions/v1beta1kind: DaemonSetmetadata:  name: node-exporter  namespace: kube-system  labels:    k8s-app: node-exporterspec:  template:    metadata:      labels:        k8s-app: node-exporter    spec:      containers:      - image: prom/node-exporter        name: node-exporter        ports:        - containerPort: 9100          protocol: TCP          name: http---apiVersion: v1kind: Servicemetadata:  labels:    k8s-app: node-exporter  name: node-exporter  namespace: kube-systemspec:  ports:  - name: http    port: 9100    nodePort: 31672    protocol: TCP  type: NodePort  selector:    k8s-app: node-exporter

Create pods and service from the above files

# kubectl create -f  

Iv. Deployment of Prometheus Components
1. rbac file

# cat rbac-setup.yaml apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  name: prometheusrules:- apiGroups: [""]  resources:  - nodes  - nodes/proxy  - services  - endpoints  - pods  verbs: ["get", "list", "watch"]- apiGroups:  - extensions  resources:  - ingresses  verbs: ["get", "list", "watch"]- nonResourceURLs: ["/metrics"]  verbs: ["get"]---apiVersion: v1kind: ServiceAccountmetadata:  name: prometheus  namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: prometheusroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: prometheussubjects:- kind: ServiceAccount  name: prometheus  namespace: kube-system

2. Managing configuration files for Prometheus components in configmap form

# cat Configmap.yaml Apiversion:v1kind:configmapmetadata:name:prometheus-config Namespace:kube-systemdata:prometh    EUS.YML: | Global:scrape_interval:15s evaluation_interval:15s scrape_configs:-job_name: ' Kubernetes-apiserve RS ' kubernetes_sd_configs:-role:endpoints Scheme:https tls_config:ca_file:/var/run/secret S/KUBERNETES.IO/SERVICEACCOUNT/CA.CRT bearer_token_file:/var/run/secrets/kubernetes.io/serviceaccount/token rel Abel_configs:-Source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_ Endpoint_port_name] action:keep regex:default;kubernetes;https-job_name: ' Kubernetes-nodes ' kube Rnetes_sd_configs:-Role:node Scheme:https tls_config:ca_file:/var/run/secrets/kubernetes.io/se      RVICEACCOUNT/CA.CRT bearer_token_file:/var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: -Action:labelMap regex: __meta_kubernetes_node_label_ (. +)-Target_label: __address__ replacement:kubernetes.defaul        T.svc:443-source_labels: [__meta_kubernetes_node_name] Regex: (. +) Target_label: __metrics_path__ Replacement:/api/v1/nodes/${1}/proxy/metrics-job_name: ' Kubernetes-cadvisor ' kubernetes_sd_configs:- Role:node Scheme:https tls_config:ca_file:/var/run/secrets/kubernetes.io/serviceaccount/ca.crt b        Earer_token_file:/var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs:-Action:labelmap      Regex: __meta_kubernetes_node_label_ (. +)-Target_label: __address__ replacement:kubernetes.default.svc:443 -Source_labels: [__meta_kubernetes_node_name] Regex: (. +) Target_label: __metrics_path__ Replac       Ement:/api/v1/nodes/${1}/proxy/metrics/cadvisor-job_name: ' Kubernetes-service-endpoints ' kubernetes_sd_configs: -Role:endpoints     Relabel_configs:-Source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] Action:kee P regex:true-source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] Action:repl      Ace Target_label: __scheme__ regex: (https?) -Source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action:replace Target_label: __m etrics_path__ regex: (. +)-source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_po RT] Action:replace Target_label: __address__ regex: ([^:]+) (?:: \d+)?;( \d+) Replacement: $1:$2-action:labelmap regex: __meta_kubernetes_service_label_ (. +)-Source_la  BELs: [__meta_kubernetes_namespace] action:replace target_label:kubernetes_namespace-source_labels: [__meta_kubernetes_service_name] action:replace target_label:kubernetes_name-job_name: ' KuBernetes-services ' Kubernetes_sd_configs:-role:service Metrics_path:/probe params:module: [Http_2xx] Relabel_configs:-source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] A Ction:keep regex:true-source_labels: [__address__] Target_label: __param_target-target_label : __address__ replacement:blackbox-exporter.example.com:9115-source_labels: [__param_target] Target_ Label:instance-action:labelmap regex: __meta_kubernetes_service_label_ (. +)-Source_labels: [__meta_k        Ubernetes_namespace] Target_label:kubernetes_namespace-source_labels: [__meta_kubernetes_service_name]      Target_label:kubernetes_name-job_name: ' kubernetes-ingresses ' kubernetes_sd_configs:-role:ingress        Relabel_configs:-Source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe] Action:keep Regex:true- Source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path] Regex: (. +);(. +);(. + ) Replacement: ${1}://${2}${3} Target_label: __param_target-target_label: __address__ Replacem Ent:blackbox-exporter.example.com:9115-source_labels: [__param_target] Target_label:instance-action : Labelmap regex: __meta_kubernetes_ingress_label_ (. +)-source_labels: [__meta_kubernetes_namespace] t Arget_label:kubernetes_namespace-source_labels: [__meta_kubernetes_ingress_name] Target_label:kubernetes_n Ame-job_name: ' Kubernetes-pods ' kubernetes_sd_configs:-role:pod relabel_configs:-Source_labe LS: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action:keep regex:true-source_labels: [ __meta_kubernetes_pod_annotation_prometheus_io_path] Action:replace Target_label: __metrics_path__ R Egex: (. +)-Source_Labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action:replace regex: ([^:]+) (?: : \d+)?;( \d+) Replacement: $1:$2 Target_label: __address__-action:labelmap regex: __meta_kubernetes_po D_label_ (. +)-source_labels: [__meta_kubernetes_namespace] Action:replace Target_label:kubernetes_na  Mespace-source_labels: [__meta_kubernetes_pod_name] Action:replace Target_label:kubernetes_pod_name

3, Prometheus deployment file

# Cat Prometheus.deploy.yml---apiversion:apps/v1beta2kind:deploymentmetadata:labels:name:prometheus-deployment Name:prometheus namespace:kube-systemspec:replicas:1 Selector:matchLabels:app:prometheus template:m Etadata:labels:app:prometheus spec:containers:-image:prom/prometheus:v2.0.0 name:p        Rometheus command:-"/bin/prometheus" args:-"--config.file=/etc/prometheus/prometheus.yml" -"--storage.tsdb.path=/prometheus"-"--storage.tsdb.retention=24h" Ports:-Containerport:9 090 protocol:tcp volumemounts:-Mountpath: "/prometheus" Name:data-mountpath: "/etc/prometheus" Name:config-volume resources:requests:cpu:100m Memory : 100Mi limits:cpu:500m memory:2500mi serviceaccountname:prometheus volum   ES:-Name:data     Emptydir: {}-Name:config-volume Configmap:name:prometheus-config        

4. Prometheus Service File

# cat prometheus.svc.yml ---kind: ServiceapiVersion: v1metadata:  labels:    app: prometheus  name: prometheus  namespace: kube-systemspec:  type: NodePort  ports:  - port: 9090    targetPort: 9090    nodePort: 30003  selector:app: prometheus

5. Create the appropriate object from the Yaml file above

# kubectl create -f  rbac-setup.yaml# kubectl create -f  configmap.yaml # kubectl create -f  prometheus.deploy.yml # kubectl create -f  

The node-exporter corresponds to a nodeport port of 31672, and by accessing http://192.168.115.5:31672/metrics you can see the corresponding metrics

The Prometheus corresponds to a nodeport port of 30003, and by accessing http://192.168.115.5:30003/target you can see that Prometheus has successfully connected K8s apiserver

You can provide basic queries on the Prometheus Web interface k8s CPU usage for each pod in the cluster, with the following query criteria:

sum by (pod_name)( rate(container_cpu_usage_seconds_total{image!="", pod_name!=""}[1m] ) )

The above query has data showing that Node-exporter writes data to Prometheus, and then we can deploy the Grafana component to achieve a more friendly WebUI presentation of the data.

V. Deployment of GRAFANA Components
1. Grafana Deployment Configuration file

# cat Grafana-deploy.yaml Apiversion:extensions/v1beta1kind:deploymentmetadata:name:grafana-core Namespace:kube-sy Stem Labels:app:grafana component:corespec:replicas:1 Template:metadata:labels:app:grafan A component:core spec:containers:-image:grafana/grafana:4.2.0 Name:grafana-core im Agepullpolicy:ifnotpresent # env:resources: # Keep request = limit to keep the container in Guar            Anteed class limits:cpu:100m Memory:100mi requests:cpu:100m Memory:100mi ENV: # The following env variables set up basic auth twith The default Admin user and a          DMin password. -name:gf_auth_basic_enabled Value: "True"-name:gf_auth_anonymous_enabled value: "False "#-Name:gf_auth_anonymous_org_role # value:admin # does not really work, because of temp LAte variables in exported dashboards: #-Name:gf_dashboards_json_enabled # Value: "True" read InessProbe:httpGet:path:/login port:3000 # initialdelayseconds:30 #      Timeoutseconds:1 volumemounts:-name:grafana-persistent-storage Mountpath:/var volumes: -Name:grafana-persistent-storage emptydir: {}

2. Grafana Service configuration file

# cat grafana-svc.yaml apiVersion: v1kind: Servicemetadata:  name: grafana  namespace: kube-system  labels:    app: grafana    component: corespec:  type: NodePort  ports:    - port: 3000  selector:    app: grafanacomponent: core3、grafana ingress配置文件# cat grafana-ing.yaml apiVersion: extensions/v1beta1kind: Ingressmetadata:   name: grafana   namespace: kube-systemspec:   rules:   - host: k8s.grafana     http:       paths:       - path: /         backend:          serviceName: grafana          servicePort: 3000

You can see that the K8s.grafana service was published successfully by visiting Traefik's WebUI

Modifying hosts parsing, accessing tests


You can also access the Nodeport port directly

The default user name and password are admin

Configure the data source to Prometheus

Import panel, you can directly enter the template number 315 online import, or download the corresponding JSON template file local import, panel template https://grafana.com/dashboards/315

After importing the panel, you can see the corresponding monitoring data.



Here to illustrate, during the testing process, import the template numbered 162, found only partial data, and the name of the pod is not friendly. Template address https://grafana.com/dashboards/162, see.

Vi. PostScript
There are some problems to follow to continue to study and solve.
1, Prometheus data storage using Emptydir. If the pod is deleted, or if the pod is migrated, Emptydir will be deleted and permanently lost. Subsequently, a Prometheus system can be configured outside the k8s cluster to permanently save the monitoring data, and the data is automatically pulled through the configuration job between the two Prometheus systems.
2, Grafana configuration data storage using Emptydir. If the pod is deleted, or if the pod is migrated, Emptydir will be deleted and permanently lost. We can also choose to configure the Grafana outside the k8s, and the data source selects the Prometheus k8s the outside of the cluster.
3, the alarm about the monitoring item (Alertmanager) has not been configured.

Refer to the documentation and thank the author for sharing!
Https://www.kubernetes.org.cn/3418.html
https://blog.qikqiak.com/post/kubernetes-monitor-prometheus-grafana/
Https://github.com/giantswarm/kubernetes-prometheus/tree/master/manifests
1190000013245394

Kubernetes Cluster Monitoring Scheme