Detailed k8s a complete monitoring scheme (HEAPSTER+GRAFANA+INFLUXDB)-kubernetes

1, analysis of the entire monitoring process

Heapster collects cluster information as a data source with k8s built-in cadvisor and summarizes valuable performance data (Metrics): CPU, memory, network traffic, and so on, and then output that data to external storage, such as INFLUXDB, Finally, it can be displayed through the corresponding UI interface, such as Grafana. In addition heapster data source and external storage are pluggable, so you can be very flexible to build a lot of monitoring solutions, such as: Heapster+elasticsearch+kibana and so on.

2. Create a K8s resource object

There are a few minor issues with the official Yml file, please refer to the following changes and instructions:

2.1. Create a Influxdb resource object

Apiversion:Apps/v1kind:Deploymentmetadata:Name:Monitoring-Influxdb namespace:Kube-Systemspec:Replicas: 1Selector:Matchlabels:Task:Monitoring k8s-App:Influxdb Template:Metadata:Labels:Task:Monitoring k8s-App:INFLUXDB Spec:Containers:      -Name:Influxdb image: k8s.GCR.io/Heapster-Influxdb-Amd64:v1.3.3Volumemounts:        -Mountpath: /dataName:influxdb-storagevolumes:-Name:influxdb-storageemptydir: {}---Apiversion:v1Kind:servicemetadata:Labels:task:monitoringKubernetes.io/cluster-Service: ' true '    kubernetes.io/name:Monitoring-Influxdb Name:Monitoring-Influxdb namespace:Kube-Systemspec:Type:Nodeport ports:  -Nodeport: 31001Port: 8086Targetport: 8086Selector:K8s-App:Influxdb

Note: Here we use Noteport expose Monitoring-influxdb service on the 31001 port of the host, then INFLUXDB server address: http://[host-ip]:31001, write down this address, To create Heapster and configure a data source for Grafana, you can use it directly.

2.1. Create a Grafana resource object

Apiversion:Apps/v1kind:Deploymentmetadata:Name:Monitoring-Grafana namespace:Kube-Systemspec:Replicas: 1Selector:Matchlabels:Task:Monitoring k8s-App:Grafana Template:Metadata:Labels:Task:Monitoring k8s-App:Grafana Spec:Containers:      -Name:Grafana image: k8s.GCR.io/Heapster-Grafana-Amd64:V4.4.3Ports:        -Containerport:  theProtocol:TCP volumemounts:        -Mountpath: /etc/ssl/certsname:ca-certificatesreadonly:true-Mountpath:/varName:Grafana-Storage env:        -Name:Influxdb_host value:Monitoring-Influxdb-Name:Gf_server_http_port value: " the"# The following env variables is required to make Grafana accessible via # the Kubernetes API-ServerProxy. on Production clusters,We recommend # Removing these env variables,Setup Auth forGrafana,and expose the Grafana # service using a loadbalancer or a Public IP.-Name:gf_auth_basic_enabled value: "false"        -Name:gf_auth_anonymous_enabled value: "true"        -Name:Gf_auth_anonymous_org_role value:Admin-Name:Gf_server_root_url # If You' re only with using the API Server proxy, set this value instead:# value: /api/v1/namespaces/kube-System/services/monitoring-Grafana/proxy value: /Volumes:      -Name:Ca-Certificates Hostpath:Path: /etc/ssl/certs-Name:grafana-storageemptydir: {}---Apiversion:v1Kind:servicemetadata:Labels:# for use as a Cluster add-on(https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)# If You is not using the this as a addon, you should comment off this line.Kubernetes.io/cluster-Service: ' true '    kubernetes.io/name:Monitoring-Grafana Name:Monitoring-Grafana namespace:Kube-Systemspec:# in a production setup,We recommend accessing Grafana through an external loadbalancer # or through a Public IP. # type:LoadBalancer # could also use Nodeport to expose the service at a randomly-Generated port type:Nodeport ports:  -Nodeport: 30108Port:  theTargetport:  theSelector:K8s-App:Grafana

Note: Here we use Noteport to expose Monitoring-grafana service on host 30108, then Grafana server address: http://registry.wuling.com:30108, accessed through the browser, Modify the data source for Grafana as follows:

The address of the INFLUXDB server, recorded in the previous step, is marked red.

2.2. Create a Heapster resource object

Apiversion:V1kind:Serviceaccountmetadata:Name:Heapster namespace:Kube-System---Apiversion:Extensions/v1beta1kind:Deploymentmetadata:Name:Heapster namespace:Kube-Systemspec:Replicas: 1Selector:Matchlabels:Task:Monitoring k8s-App:Heapster Template:Metadata:Labels:Task:Monitoring k8s-App:Heapster Spec:Serviceaccountname:Heapster containers:      -Name:Heapster image: k8s.GCR.io/Heapster-Amd64:v1.4.2Imagepullpolicy:Ifnotpresent command:        - /heapster---source=kubernetes:https://kubernetes.default---sink=influxdb:http://150.109.39.33:31001 # Here fill in the address of the INFLUXDB server just recorded. ---Apiversion:v1Kind:servicemetadata:Labels:task:monitoring# for use as a Cluster add-on(https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)# If You is not using the this as a addon, you should comment off this line.Kubernetes.io/cluster-Service: ' true '    kubernetes.io/name:Heapster Name:Heapster namespace:Kube-Systemspec:Ports:  -Port:  theTargetport: 8082Selector:K8s-App:Heapster

--SOURCE Specifies the data source for Heapster to obtain the cluster information. Reference: HTTPS://GITHUB.COM/KUBERNETES/HEAPSTER/BLOB/MASTER/DOCS/SOURCE-CONFIGURATION.MD
--sink for Heaster specify back-end storage, here we use Influxdb, others, please refer to: https://github.com/kubernetes/heapster/blob/master/docs/sink-owners.md
here Heapster left a pit, please continue to look down when I deploy the Heapster, by looking at the mirror of the Heapster container group to discover:

Many people think is the problem of HTTPS or k8s configuration, so go to the flurried to configure the insecure HTTP way, resulting in more and more deep pits, more and more low transparency, it is impossible, I also get a long time, are more than retwist, here omit 10,000 words ... , when these paths have been traveled, re-read the following text:

Only to find the issue of permissions, Heaster by default with a token (token) and apiserver authentication, through the view heapster.yml found serviceaccountname:heapster , now understand, Is Heaster no permissions, then how to authorize it----- to heaster bind a privileged role on the line , as follows:

apiVersion:rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: heapsterroleRef:  apiGroup:rbac.authorization.k8s.io  kind: ClusterRole  name: cluster-adminsubjects:- kind: ServiceAccount  name: heapster  namespace: kube-system

When creating Heapster resources, add the code directly, and it's OK.

3. View monitoring details 3.1, through dashboard View cluster overview

After successful deployment of the entire monitoring solution, it can be seen that the specific CPU and memory utilization of the object can be rendered on the dashboard in different granularity/dimensions.

3.2. View cluster details via Grafana (CPU, memory, filesystem)

Detailed k8s a complete monitoring scheme (HEAPSTER+GRAFANA+INFLUXDB)-kubernetes