Metrics and Alerting with Prometheus, Grafana, and Alertmanager

What you'll build and why

Prometheus scrapes metrics from your hosts and containers on a schedule and stores them as time series; Grafana turns that data into dashboards; Alertmanager turns specific conditions in that data into actual notifications. Together, this is the current de facto standard homelab (and production) observability stack -- widely documented, and the skills transfer directly if you ever do this professionally.

Don't build this if: you just want to know "is this one service up," not ongoing metrics -- Uptime Kuma (mentioned in this section's Reference page) is a much lighter tool for that specific, narrower question. This guide is for when you want to understand how your infrastructure is behaving over time, not just whether it's currently reachable.

How it works

  node_exporter (on each host)  --  exposes host metrics
cAdvisor (on each Docker host)  --  exposes container metrics
│                    │
│  Prometheus pulls (scrapes) from both on a schedule
▼                    ▼
Prometheus  --  stores metrics as time series, evaluates
│          alert rules against them continuously
│
├──► Grafana  --  queries Prometheus, renders dashboards
│
└──► Alertmanager  --  receives firing alerts from
Prometheus, routes/deduplicates/
notifies (email, webhook, etc.)

The pull model matters: Prometheus decides when to scrape each target, rather than every target needing to know where to push data. This makes adding a new host mostly a matter of running an exporter on it and adding one line to Prometheus's scrape config.

Before you start

Decision: what to monitor first? Don't try to instrument everything on day one. Host-level metrics (CPU, memory, disk) via node_exporter on every machine, plus container metrics via cAdvisor on your Docker hosts, is a solid, genuinely useful starting scope -- application-specific metrics (a service's own request rates, error counts) can come later once the basics are running.

Steps

Step 1: Run node_exporter on each host you want metrics from

services:
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
network_mode: host
pid: host
volumes:
- "/:/host:ro,rslave"
command:
- '--path.rootfs=/host'
restart: unless-stopped

Repeat on every host whose metrics you want -- one node_exporter per host, each exposing on port 9100.

Step 2: Run cAdvisor on your Docker hosts

services:
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- "8080:8080"
volumes:
- "/:/rootfs:ro"
- "/var/run:/var/run:ro"
- "/sys:/sys:ro"
- "/var/lib/docker:/var/lib/docker:ro"
restart: unless-stopped

Step 3: Run Prometheus, configured to scrape both

services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- "9090:9090"
volumes:
- "./prometheus.yml:/etc/prometheus/prometheus.yml"
- "prometheus_data:/prometheus"
restart: unless-stopped
volumes:
prometheus_data:

prometheus.yml:

global:
scrape_interval: 15s
scrape_configs:
- job_name: 'node'
static_configs:
- targets: ['<host-ip>:9100']
- job_name: 'cadvisor'
static_configs:
- targets: ['<host-ip>:8080']

Add one targets entry per host as you expand.

Step 4: Run Grafana and connect it to Prometheus

services:
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
volumes:
- "grafana_data:/var/lib/grafana"
restart: unless-stopped
volumes:
grafana_data:

Visit http://<host-ip>:3000 (default login admin/admin, changed on first login), add Prometheus as a data source (Connections → Data sources → Add, URL http://<prometheus-host>:9090), and import a community node_exporter/cAdvisor dashboard (Grafana's dashboard library has well-maintained ones for both) rather than building from scratch.

Step 5: Run Alertmanager and configure Prometheus to use it

services:
alertmanager:
image: prom/alertmanager:latest
container_name: alertmanager
ports:
- "9093:9093"
volumes:
- "./alertmanager.yml:/etc/alertmanager/alertmanager.yml"
restart: unless-stopped

Add to prometheus.yml:

alerting:
alertmanagers:
- static_configs:
- targets: ['<alertmanager-host>:9093']
rule_files:
- "alerts.yml"

alerts.yml (a real, useful first alert):

groups:
- name: basic
rules:
- alert: HostDown
expr: up == 0
for: 5m
annotations:
summary: "{{ $labels.instance }} has been down for 5+ minutes"

alertmanager.yml needs a receiver configured (email, webhook, or a service like Pushover) -- see Alertmanager's own configuration documentation for the exact syntax for your chosen notification method.

Verify it works

• http://<prometheus-host>:9090/targets shows every configured target as UP
• Grafana's imported dashboard shows real, updating data for at least one host
• http://<alertmanager-host>:9093 loads and shows no active alerts (assuming everything's healthy)
• Failure test: stop node_exporter on one host (docker stop node-exporter). Within the for: 5m window configured in Step 5's alert rule, confirm it fires in Prometheus (Alerts page shows it as firing) and that Alertmanager actually sends the configured notification -- not just that the rule exists, that the whole path to an actual notification works

Secure it

• What's exposed: Prometheus, Grafana, and Alertmanager's web UIs should all be LAN-only by default in this guide -- none should be port-forwarded to the internet without additional authentication in front of them (see the SSO guide elsewhere in this section for adding that properly).
• Default credentials: Grafana's default admin/admin login must be changed on first use -- don't skip this.
• Metric data sensitivity: be aware that detailed host/container metrics can reveal information about what you're running and when (usage patterns) -- not usually sensitive for a homelab, but worth being deliberate about who has access to Grafana if that's ever a concern.

Back it up and maintain it

What matters: Grafana's own data volume (dashboards, data source config, users) and your Prometheus/Alertmanager config files -- all small and worth version-controlling in git. Prometheus's actual metric data (prometheus_data) is large and, for most homelab purposes, not worth backing up -- it's historical trend data, not irreplaceable state.

Update cadence: all three components ship regularly; update on your own schedule, watching for breaking config-format changes in release notes (rare, but they happen).

What to monitor: the stack monitoring itself is somewhat recursive -- at minimum, periodically check the /targets page for any target that's silently gone DOWN and been ignored.

Troubleshooting

Logs: docker logs prometheus, docker logs grafana, docker logs alertmanager.

Undo

docker compose down removes all four containers; delete their volumes to remove all stored data/config. Remove any port forwards or reverse proxy entries you'd added for external access (which this guide didn't recommend in the first place).

Go further

• Centralized Logging with Grafana Loki -- the logs complement to this guide's metrics, in the same Grafana dashboard
• Add more exporters as needed (SNMP exporter for network gear, a service's own Prometheus-format metrics endpoint if it has one)
• Explore Grafana's own alerting (in addition to or instead of Alertmanager) if you want alert rules defined closer to the dashboards themselves

Resources

Official documentation:

• Prometheus documentation -- getting started and scrape config, verified this session
• Grafana documentation
• Alertmanager documentation

Source and releases:

• prometheus/prometheus releases
• grafana/grafana releases

Community:

• Prometheus community
• r/grafana

Related DaemonPress projects:

• Centralized Logging with Grafana Loki
• Managing a Homelab with Docker Compose, Portainer, and Watchtower

Last verified: 2026-09-21, checked against official Prometheus documentation.