infra-monitoring
Configure or audit AICP's monitoring stack — Prometheus (`:9090`) + Grafana (`:3000`) + 7 alert rules in `config/alerts.yaml` + AICP's own `:9101/metrics` exporter + LocalAI's built-in `:8090/metrics`
SKILL.md.claude/skills/infra-monitoring/SKILL.md
GitHubinfra-monitoring
Configures or audits AICP's monitoring stack. AICP ships a complete
observability layer (Prometheus + Grafana behind a Docker compose profile +
7 baseline alerts + per-component metrics endpoints). This skill is for
working WITH the existing stack — adding/tuning alerts, reviewing dashboards,
verifying metrics flow, debugging missing signals — not for replacing the
stack (replacing would be architecture-propose scope).
Trigger phrases (when to load this skill)
Load when the conversation matches any of:
- Direct verb: operator says "set up monitoring", "add an alert", "tune alert X", "what dashboards exist", "audit observability", "monitoring isn't telling me X", "where's the metric for Y"
- Stage 4 reliability work: AICP's reliability profile (circuit breaker + warmup + DLQ + reports) emits metrics this skill maintains the dashboards for
- Post-incident retrospective: an incident showed a missing signal — what alert SHOULD have fired? Add it.
- Pre-fleet-rollout: before AICP serves a fleet, confirm metrics flow + alerts are live + on-call routing works
- Periodic audit: quarterly review of which alerts fired, which were noise, which never fired (silent gaps)
Do NOT load when:
- The concern is correctness (load
feature-testorsystematic-debugging) - The concern is a live incident (load
ops-incidentfirst; monitoring tuning is post-incident) - Setting up tests + coverage (load
foundation-testing) - Setting up logging from scratch (load
foundation-logging— different layer; logs feed metrics but they're separate skills) - Replacing Prometheus/Grafana with a different stack (load
architecture-propose)
Operations
This skill has 4 named operations. Execute in order.
Operation 1: Verify the monitoring stack is running + reachable
Trigger: skill loaded; monitoring work requested.
Process:
Confirm the Docker compose monitoring profile is up:
docker compose ps 2>&1 | grep -E 'prometheus|grafana' || echo "Monitoring stack not running"If down, start it:
make monitoring-up(Prometheus on:9090, Grafana on:3000admin/aicp).Verify each metrics endpoint is scrapable:
curl -sf localhost:9101/metrics | head -5 # AICP's own metrics curl -sf localhost:8090/metrics | head -5 # LocalAI built-in metrics curl -sf localhost:9090/-/healthy # Prometheus self curl -sf localhost:3000/api/health # Grafana selfAny failure: stop and report. Monitoring config is moot if endpoints are unreachable.
Confirm Prometheus is scraping all targets:
curl -s localhost:9090/api/v1/targets | python3 -c "import json,sys; \ d=json.load(sys.stdin); print('\n'.join(f\"{t['labels']['job']}: {t['health']}\" \ for t in d['data']['activeTargets']))"Each target should be
up.downtargets mean Prometheus isn't getting metrics — common cause: Docker network mismatch or wrong port.List existing dashboards in Grafana:
curl -s -u admin:aicp localhost:3000/api/search?type=dash-db \ | python3 -c "import json,sys; print('\n'.join(d['title'] for d in json.load(sys.stdin)))"Read
config/alerts.yamlto capture the 7 baseline alerts: stuck model, latency, errors, swaps, quality, cost, memory. Note their thresholds.Write findings to
wiki/decisions/00_inbox/monitoring-audit-<date>.md(type=reference). Include: stack health, target health, dashboard inventory, alert inventory.
Quality bar (Operation 1 done when):
- All 4 monitoring components confirmed running (or started + verified)
- Each metrics endpoint reachable
- Prometheus targets all
up(or down targets investigated) - Dashboards listed
- 7 baseline alerts inventoried with thresholds
- Audit page created
Operation 2: Identify the gap (what to add or tune)
Trigger: Operation 1 baseline established; operator stated the gap.
Process:
- Frame the gap concretely. NOT "monitoring is incomplete" but ONE of:
- Missing metric: a KPI exists in code (e.g., circuit breaker state transitions) but isn't exposed via
:9101/metrics - Missing alert: a metric IS exposed but no alert fires when it crosses a threshold
- Noisy alert: an alert fires too often (false positives degrade trust)
- Silent alert: an alert never fires (threshold too lax OR the underlying metric isn't moving)
- Missing dashboard: data exists but isn't visualized for human consumption
- Missing metric: a KPI exists in code (e.g., circuit breaker state transitions) but isn't exposed via
- For each gap type, the FIX shape is different:
- Missing metric → modify
aicp/core/prometheus.py(the MetricsCollector) to expose the new counter/gauge/histogram. Then add a Prometheus scrape config if it lives at a new endpoint. - Missing alert → add to
config/alerts.yaml. Define expression (PromQL), severity, threshold, alerting message. - Noisy alert → tune threshold. Capture the past N firings (
grep ALERT /var/log/...or query Alertmanager) and pick a threshold above the noise floor. - Silent alert → either lower threshold OR fix the underlying metric (it may not be moving because the code path isn't exercised).
- Missing dashboard → author a Grafana JSON or use the UI to build, then export.
- Missing metric → modify
- For each fix, identify the BLAST RADIUS: who depends on this signal? Affects only solo dev? Affects fleet routing? Affects on-call routing? More dependents = more careful change.
Quality bar (Operation 2 done when):
- Gap framed as ONE of the 5 categories above (not vague "improve monitoring")
- Fix shape identified per the category
- Blast radius assessed
- Operator approved the fix scope before authoring
Operation 3: Author the change (alert / metric / dashboard)
Trigger: Operation 2 plan approved.
Process:
For alert changes — edit
config/alerts.yaml. Each alert has the structure (per existing 7 examples):- alert: <PascalCase name> expr: <PromQL expression> for: <duration before firing, e.g., 5m> labels: severity: warning # or critical annotations: summary: <one-line> description: <multi-line context with mitigation hint>New alerts should follow the existing 7's pattern. Don't invent new label schemas.
For metric changes — edit
aicp/core/prometheus.py. Add the new counter/gauge/histogram toMetricsCollector. Wire in the call site (the code path that should increment). Per feature-implement Gotcha 1: NO ORPHAN. New metric must be incremented by some real code path.For dashboard changes — Grafana UI to build, then
Settings → JSON Model → Save to file. Commit the JSON tomonitoring/dashboards/(create if missing). Provision via Grafana'sprovisioning/dashboards/*.yaml.After EACH change:
- Re-run Operation 1 verifications (endpoints reachable, targets up)
- For new metric: confirm it appears at
:9101/metricsand Prometheus has scraped it (curl 'localhost:9090/api/v1/query?query=<metric>') - For new alert: confirm the alert appears in Prometheus's rules (
curl 'localhost:9090/api/v1/rules') - For new dashboard: load it in Grafana, verify panels render data
Run AICP's full test suite to ensure metric wiring didn't break anything:
pytest tests/ -x --tb=short.Commit with conventional format:
feat(monitoring): add <X>orchore(monitoring): tune <X>.
Quality bar (Operation 3 done when):
- Change applied per the category-specific shape
- Verification ran AFTER the change (not just trusted)
- No orphan metric (every new metric has a code path that emits it)
- AICP test suite passes (no test regression)
- Committed in conventional format
Operation 4: Document + close out
Trigger: Operation 3 change landed.
Process:
- Update the audit page from Operation 1 with the change applied. Include the alert/metric/dashboard name and brief description.
- If the gap surfaced a SYSTEMIC pattern (e.g., "all backend errors lacked granular metric breakdowns"), contribute back as a lesson:
gateway contribute --type lesson. - If the alert needs runbook entries (when it fires, what does on-call do?), update or create the runbook in
docs/runbooks/with: trigger condition, immediate mitigation, root-cause investigation steps, escalation path. - Run
tools/lint.pyon any wiki content authored. - Inform operator: change live, runbook updated, audit page captured.
Quality bar (Operation 4 done when):
- Audit page reflects the change
- Runbook updated if the alert needs operational response
- Lesson contributed if systemic
- Wiki lint passes
Gotchas (known failure modes — read before doing)
Gotcha 1: Adding metrics without alerts (silent capability)
The temptation: add a metric, ship it, "we can alert on it later." NO — a metric without an alert is data that nobody sees until they go looking. Per Quality Standards principle: if it's worth measuring, it's worth alerting on (with appropriate threshold).
Detection: did Operation 3 add a metric without a corresponding alert added in the same batch?
The rule: every NEW metric ships with at least an "informational" alert (low severity, high threshold) so it appears in the alert inventory. Tuning the threshold later is fine; having no alert is silent.
Gotcha 2: Alert thresholds set to "never fire" (alert theater)
The temptation: set the threshold so high it never fires (e.g., latency > 60s). It's "always green" so it looks healthy. NO — an alert that never fires means you've added work for the alert system but no signal for the operator. Per Quality Standards anti-pattern: this is "compliance theater for alerts."
Detection: review the alert's PromQL expression — would it fire under normal degraded conditions?
The rule: thresholds calibrated to actual data. Use the past N days of metrics to find the p95 baseline; set the threshold at p99 + headroom. Never-fires thresholds get flagged in periodic alert review.
Gotcha 3: Noisy alerts that get muted (worse than no alert)
A noisy alert (fires constantly) gets muted by on-call. Eventually a REAL incident triggers it; nobody sees because the alert was muted. Per Quality Standards principle: false positives kill trust.
Detection: check Alertmanager (or Prometheus alert UI) for muted/silenced alerts. Each mute is a sign the alert is broken.
The rule: noisy alerts get FIXED, not muted. Either tune the threshold or fix the underlying flaky behavior. Mutes are short-term (during an incident); persistent mutes are technical debt.
Gotcha 4: Dashboards without the queries (Grafana lock-in)
The temptation: build dashboards in the Grafana UI, save them. The dashboard JSON lives only in Grafana's database — if Grafana restarts cleanly, dashboards are gone. Per CLAUDE.md "IaC only" rule: dashboards should be reproducible via make setup.
Detection: are dashboards in version control? ls monitoring/dashboards/*.json 2>/dev/null should match Grafana's dashboard list.
The rule: dashboards are IaC. JSON in monitoring/dashboards/, provisioned via Grafana's provisioning config. UI is for AUTHORING; the file is the source of truth.
Gotcha 5: Auditing alerts vs auditing actual firings
The temptation: review config/alerts.yaml and report "we have 7 alerts, all reasonable." But the question wasn't "do alerts exist?" — it was "do alerts work?" Reviewing the YAML doesn't reveal which alerts fired vs which were silent.
Detection: did Operation 1 step 5 capture only the alert DEFINITIONS, or also their FIRING HISTORY (last N days)?
The rule: alert audits include firing history. An alert that hasn't fired in 90 days is suspicious (silent gap or threshold too lax). Use Prometheus query history or Alertmanager's alert log.
Reference exemplars
Per Extension Standards, reference exemplars are the second brain's model-builder and wiki-agent skills. For the OUTPUT artifact:
- Real alerts file: see config/alerts.yaml — the 7 baseline alerts that this skill maintains/extends
- Real metrics exporter: see aicp/core/prometheus.py — the MetricsCollector that emits AICP-specific metrics
- Monitoring audit page shape: same as sibling quality audits (coverage / lint / debt / performance) — reference page with per-component findings + recommendations
Domain context
This skill operates in backend-ai-platform-python — see wiki/config/domain-profiles/backend-ai-platform-python.yaml. AICP-specific monitoring stack: Prometheus :9090, Grafana :3000 (admin/aicp), AICP :9101/metrics, LocalAI :8090/metrics. Stack starts/stops via make monitoring-up / make monitoring-down.
Related skills
| Skill | When | Why distinct |
|---|---|---|
| infra-security | security audit | Different observability dimension (security events vs operational metrics) |
| quality-performance | perf benchmarks | Bench is point-in-time; this skill is continuous monitoring |
| ops-incident | live incident response | Reactive; this skill is proactive (alerts trigger ops-incident) |
| ops-deploy | deployment with checks | Deploy uses health checks (this skill ensures those checks have signals) |
| foundation-logging | structured logging setup | Logs are different from metrics; complementary |
| infra-search | search infrastructure | Different infrastructure category |