Exploration of kubernetes resource monitoring

Build kubernetes cluster, there is a default dashboard, but this dashboard is relatively humble, can not be customized display. So we're going to use GRAFANA+HEAPSTER+INFLUXDB to build an integrated monitoring platform.

What's the first thing? Get the source code from Https://github.com/kubernetes/heapster, in the Heapster/deploy/kube-config/influxdb folder will all Yaml files, use Kubectl Create, so that you can separate the

Heapster,grafana,influxdb three different types of pods.

More important: The configuration of--source and--sink in Heapster.yaml:

Https://github.com/nefuddos/heapster/blob/master/docs/sink-configuration.md

Https://github.com/nefuddos/heapster/blob/master/docs/source-configuration.md

The problem encountered:

Because Heapster collects the cluster information for some time, the CPU and mem information seen on the dashboard is slower.

There is one more error log:

2017-09-16t02:45:40.136702919z E0916 02:45:40.136411 1 influxdb.go:150] Failed to create infuxdb:failed to ping influxdb Server at ' monitoring-influxdb:8086 '-Get http://monitoring-influxdb:8086/ping:dial tcp:lookup monitoring-influxdb on 10.254.0.2:53:server misbehaving

According to the following information, dial TCP xxx, indicating that Heapster is located in the pod access kube-dns the problem occurs. Check that all POD nodes are inaccessible on the machine where the Kube-dns is located.

The Kube-dns.yaml is then manually set up on a node that can be accessed.

Heapster structure diagram:

Kubernetes Resource monitoring exploration