Monitoring Docker Containers with cAdvisor and Prometheus

Why Container Monitoring Matters

Running Docker containers without monitoring is like driving without a dashboard. You need to know CPU usage, memory consumption, network I/O, and disk usage for every container — in real-time and historically.

The Monitoring Stack

Our monitoring stack consists of three components:

• cAdvisor: Collects container metrics (built by Google)
• Prometheus: Stores metrics and evaluates alerting rules
• Grafana: Visualizes everything in beautiful dashboards

Docker Compose Setup

version: "3.8"
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    privileged: true
    devices:
      - /dev/kmsg:/dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/alert-rules.yml:/etc/prometheus/alert-rules.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: your-grafana-password
      GF_SERVER_ROOT_URL: https://monitoring.example.com
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    ports:
      - "3000:3000"
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

Prometheus Configuration

# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert-rules.yml"

scrape_configs:
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "docker"
    static_configs:
      - targets: ["host.docker.internal:9323"]

Alert Rules

# prometheus/alert-rules.yml
groups:
  - name: container-alerts
    rules:
      - alert: ContainerHighCPU
        expr: rate(container_cpu_usage_seconds_total{name!=""}[5m]) * 100 > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ .Labels.name }} high CPU usage"
          description: "Container {{ .Labels.name }} CPU usage is above 80% for 5 minutes"

      - alert: ContainerHighMemory
        expr: container_memory_usage_bytes{name!=""} / container_spec_memory_limit_bytes{name!=""} * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ .Labels.name }} high memory usage"

      - alert: ContainerDown
        expr: absent(container_last_seen{name=~".+"})
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Container {{ .Labels.name }} is down"

      - alert: HighDiskUsage
        expr: (node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_avail_bytes{mountpoint="/"}) / node_filesystem_size_bytes{mountpoint="/"} * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk usage above 85%"

Useful PromQL Queries

# CPU usage per container (percentage)
rate(container_cpu_usage_seconds_total{name!=""}[5m]) * 100

# Memory usage per container (MB)
container_memory_usage_bytes{name!=""} / 1024 / 1024

# Network received per container (bytes/sec)
rate(container_network_receive_bytes_total{name!=""}[5m])

# Network transmitted per container (bytes/sec)
rate(container_network_transmit_bytes_total{name!=""}[5m])

# Disk read per container (bytes/sec)
rate(container_fs_reads_bytes_total{name!=""}[5m])

# Container restart count
increase(container_restart_count{name!=""}[1h])

# Top 10 containers by memory usage
topk(10, container_memory_usage_bytes{name!=""})

Grafana Dashboard

Import the Docker Container dashboard (ID: 11600) from grafana.com for a pre-built view. For custom dashboards, use these panels:

Container Overview Panel

{
  "title": "Container CPU Usage",
  "type": "timeseries",
  "datasource": "Prometheus",
  "targets": [
    {
      "expr": "rate(container_cpu_usage_seconds_total{name!=''}[5m]) * 100",
      "legendFormat": "{{ name }}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percent",
      "thresholds": {
        "steps": [
          { "value": 0, "color": "green" },
          { "value": 50, "color": "yellow" },
          { "value": 80, "color": "red" }
        ]
      }
    }
  }
}

Alertmanager Integration

Route Prometheus alerts to Ntfy for push notifications:

# alertmanager.yml
route:
  receiver: "ntfy"
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: "ntfy"
    webhook_configs:
      - url: "https://notify.example.com/ops-alerts"
        send_resolved: true