Deploy Cadvisor + InfluxDB + grafana Docker monitoring on marathon

Google Cadvisor is a great tool for monitoring Docker containers, but it only shows real-time data by default and does not store historical data. To store and display historical data, custom displays, you can integrate Cadvisor with Influxdb, Grafana.

Required Image:

Tutum/influxdb

Google/cadvisor

Grafana/grafana

Installation: There are 8 mesos-slave that need to run a Cadvisor Docker instance to monitor Docker, then run a Influxdb Docker instance to store cadvisor generated data, then run a Grafana The Docker instance extracts the data from the INFLUXDB database and displays it on top of the graph.

Deploying InfluxDB

• Influxdb only need one instance;
• The UI is published via MARATHON-LB's virtual host;
• Data port 8086 is published via Serviceport to the slaves where marathon-lb is located;
• Serviceport need to be set to a fixed value, such as: 28086, in order to cadvisor and Grafana connection;
• Data Catalog/data mapped to NFS shared directory;

{
  "id": "/influxdb",
  "cmd": null,
  "cpus": 0.3,
  "mem": 1024,
  "disk": 0,
  "instances": 1,
  "acceptedResourceRoles": [],
  "container": {
    "type": "DOCKER",
    "volumes": [
      {
        "containerPath": "/etc/localtime",
        "hostPath": "/etc/localtime",
        "mode": "RO"
      },
      {
        "containerPath": "/data",
        "hostPath": "/home/nfs/InfluxDB",
        "mode": "RW"
      }
    ],
    "docker": {
      "image": "10.80.163.110:5000/influxdb",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8083,
          "hostPort": 0,
          "servicePort": 10010,
          "protocol": "tcp",
          "labels": {}
        },
        {
          "containerPort": 8086,
          "hostPort": 0,
          "servicePort": 28086,
          "protocol": "tcp",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": false
    }
  },
  "labels": {
    "HAPROXY_0_VHOST": "influxdb.osp.cloud",
    "HAPROXY_GROUP": "external"
  },
  "portDefinitions": [
    {
      "port": 10010,
      "protocol": "tcp",
      "name": "default",
      "labels": {}
    },
    {
      "port": 28086,
      "protocol": "tcp",
      "labels": {}
    }
  ]
}

Creating a monitoring database

Open Http://influxdb.osp.cloud set Host and Port to Http://influxdb.osp.cloud and 28086, respectively.

Create a separate database for each mesos slave, respectively: cadvisor112, cadvisor113, ...

Deploying Cadvisor

• Each mesos slave to deploy an instance;
• The UI is published via MARATHON-LB's virtual host;
• Set Storage_drive to Influxdb;

{
  "id": "/cadvisor112",
  "cmd": null,
  "cpus": 0.1,
  "mem": 256,
  "disk": 0,
  "instances": 1,
  "constraints": [
    [
      "hostname",
      "CLUSTER",
      "10.80.163.112"
    ]
  ],
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "volumes": [
      {
        "containerPath": "/etc/localtime",
        "hostPath": "/etc/localtime",
        "mode": "RO"
      },
      {
        "containerPath": "/rootfs",
        "hostPath": "/",
        "mode": "RO"
      },
      {
        "containerPath": "/var/run",
        "hostPath": "/var/run",
        "mode": "RW"
      },
      {
        "containerPath": "/sys",
        "hostPath": "/sys",
        "mode": "RO"
      },
      {
        "containerPath": "/var/lib/docker",
        "hostPath": "/var/lib/docker",
        "mode": "RO"
      },
      {
        "containerPath": "/cgroup",
        "hostPath": "/cgroup",
        "mode": "RO"
      }
    ],
    "docker": {
      "image": "10.80.163.110:5000/cadvisor",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 0,
          "servicePort": 10011,
          "protocol": "tcp",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": false
    }
  },
  "labels": {
    "HAPROXY_0_VHOST": "cadvisor112.zyyt.osp.cloud",
    "HAPROXY_GROUP": "external"
  },
  "portDefinitions": [
    {
      "port": 10011,
      "protocol": "tcp",
      "name": "default",
      "labels": {}
    }
  ],
  "args": [
    "-storage_driver",
    "influxdb",
    "-storage_driver_host",
    "influxdb.osp.cloud:28086",
    "-storage_driver_db",
    "cadvisor112"
  ]
}

To view the Cadvisor UI:

Http://cadvisor112.zyyt.osp.cloud

Deploying Grafana

• Only one instance needs to be deployed;
• UI is published via MARATHON-LB virtual host;
• Data Catalog/var/lib/grafana mapped to NFS shared storage for easy persistence of storage;

{
  "id": "/grafana",
  "cmd": null,
  "cpus": 0.3,
  "mem": 512,
  "disk": 0,
  "instances": 1,
  "constraints": [
    [
      "hostname",
      "CLUSTER",
      "10.80.163.112"
    ]
  ],
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "volumes": [
      {
        "containerPath": "/etc/localtime",
        "hostPath": "/etc/localtime",
        "mode": "RO"
      },
      {
        "containerPath": "/var/lib/grafana",
        "hostPath": "/home/nfs/GrafanaData",
        "mode": "RW"
      }
    ],
    "docker": {
      "image": "10.80.163.110:5000/grafana:2.0.2",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 3000,
          "hostPort": 0,
          "servicePort": 10012,
          "protocol": "tcp",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": false
    }
  },
  "labels": {
    "HAPROXY_0_VHOST": "grafana.zyyt.osp.cloud",
    "HAPROXY_GROUP": "external"
  },
  "portDefinitions": [
    {
      "port": 10012,
      "protocol": "tcp",
      "name": "default",
      "labels": {}
    }
  ]
}

Create a data analysis diagram

To open the Grafana UI:

Http://grafana.zyyt.osp.cloud

To set up a data source:

• Type: InfluxDB
• url:http://influxdb.osp.cloud:28086
• Access:direct
• Database: Select a slave, such as: cadvisor112

Create graph:

：

Reference: http://www.mamicode.com/info-detail-1393800.html

Deploy Cadvisor + InfluxDB + grafana Docker monitoring on marathon