dgx-spark-observability

Layer-1 host observability for DGX Spark / Blackwell GB10 / sm_121a / aarch64.

A self-contained, 11-container LGTM-style stack:

• Prometheus + Grafana + Loki + Promtail + Alertmanager + ntfy (self-hosted push).
• Exporters: node-exporter, smartctl-exporter (arm64-rebuilt), cAdvisor, dcgm-exporter, plus an nvidia-textfile sidecar that fills DCGM gaps on Blackwell unified memory.
• 6-month metric retention, 90-day system-log retention.
• 13 hardware alerts (RAM/disk/GPU temp/throttle/OOM/NVMe SMART/etc.) with runbook annotations.
• Overnight push mute (23:00–08:00 in ${TZ}).
• Custom 38-panel at-a-glance host dashboard (Spark Host Overview) plus three pre-wired Grafana community dashboards.

Generic LGTM templates assume x86_64 + discrete-VRAM NVIDIA GPUs. DGX Spark is aarch64 + Blackwell GB10 with unified memory, and that combination breaks several common assumptions:

• smartctl_exporter ships amd64-only manifests upstream.
• dcgm-exporter 4.x doesn't expose FB_USED/FB_FREE on unified memory and doesn't load the profiling module on sm_121a.
• cadvisor v0.49.x uses a Docker client API too old for Docker 25+.
• Alertmanager rejects overnight mute windows directly.
• A handful of nvidia-smi query fields behave differently on unified memory and on driver 580+.

This template documents and works around all of these. See docs/DGX-SPARK-QUIRKS.md for the full list with symptom / cause / fix / upstream status.

git clone https://github.com/cadaverine/dgx-spark-observability.git
cd dgx-spark-observability

cp .env.example .env
# Edit .env: set GRAFANA_ADMIN_PASSWORD, NTFY_TOPIC (must be unguessable),
#            BIND_ADDRESS, TZ.

docker network create observability-net
docker compose up -d
# Wait ~60s for everything to be ready.

curl -sf http://127.0.0.1:3001/api/health   # Grafana check

# Optional: download three Grafana community dashboards
./scripts/fetch-community-dashboards.sh
docker compose restart grafana

38-panel dashboard covering system / GPU / thermal / storage / network / containers + logs in one view. See grafana/dashboards/custom/spark-host-overview.json.

All UIs bind to ${BIND_ADDRESS} (default 127.0.0.1). For remote access either set BIND_ADDRESS to a Tailscale IP, or SSH-forward:

See docs/ARCHITECTURE.md for the full design — network model, metric sources, label schema, log pipeline, dashboard structure, alerting design, retention choices.

See docs/DGX-SPARK-QUIRKS.md for the Blackwell / aarch64 specific gotchas this template handles.

The template ships with two static labels — host=spark, hardware=gb10 — applied in three places:

• prometheus/prometheus.yml (external_labels + per-job instance)
• promtail/promtail-config.yml (per-job labels)
• Some alert annotations in prometheus/rules/host.yml

If you're running on something other than a DGX Spark, search-and-replace spark and gb10 to whatever fits your inventory. The Grafana dashboard's $host template variable picks them up automatically.

This template is intentionally host-level only. If you also want application-level monitoring, fork or extend:

• vLLM /metrics scrape (TTFT, KV cache, queue, throughput).
• LiteLLM / per-key cost tracking.
• Container stdout/stderr shipping to Loki.
• Distributed tracing (Tempo, etc.).
• Multi-host federation, HA replicas, SSO.

The label schema and Prometheus / Loki are deliberately not pre-bound to any application so that adding these on top does not require rewriting Layer 1.

PRs welcome — particularly for additional hardware compatibility (Jetson, Grace+Hopper) or further quirk documentation.

License: MIT.