observability-checklist

What good observability looks like — structured logs, trace spans, metrics, correlation ids, and the instrumentation rules that turn logs into answers. Use when adding a feature, reviewing a service,

SKILL.md

GitHub

Observability checklist

The goal isn't logs — it's answering "what happened" in under five minutes at 2am. Design backward from the question.

Logs

Structured or nothing

  • JSON (or logfmt), not free text.
  • Every log has: ts, level, service, request_id, msg, and relevant fields.
  • Fields are snake_case or camelCase — pick one per service. No mixing.
  • Durations in milliseconds as numbers, not "took 2s" strings.

Levels are a contract

  • error: something failed the user can't recover from. Alert-worthy.
  • warn: something unexpected but handled. Review-worthy.
  • info: the shape of normal traffic. Sampling OK.
  • debug: developer-only. Off in prod.
  • Don't invent new levels.

What to log

  • Request entry (method, path, user, request_id).
  • Outcome (status, duration).
  • Branch decisions that matter (authz check, cache hit/miss, retry).
  • Errors with the full context you need to reproduce.

What NOT to log

  • Secrets, tokens, passwords, PII — scrub before logging.
  • The full request body by default. Specific fields, yes. Everything, no.
  • Stack traces as separate lines — include as a stack field so they stay with the event.

Traces

Span per unit of work

  • HTTP request handler = 1 span.
  • Outbound call = child span.
  • DB query = child span (with db.statement truncated / hashed, not raw if PII risk).
  • Background job step = 1 span.

Mandatory attributes

  • service.name, service.version.
  • For HTTP: http.method, http.route, http.status_code.
  • For DB: db.system, db.operation.
  • For errors: exception.type, exception.message.

Propagation

  • Every outbound request carries trace context headers.
  • Every background job enqueues with trace context.
  • A trace that ends at the HTTP edge and restarts at the worker is useless.

Metrics

RED for services

  • Rate — requests per second, per endpoint.
  • Errors — error rate, per endpoint.
  • Duration — p50/p95/p99, per endpoint.

USE for resources

  • Utilization — % of capacity in use.
  • Saturation — queue depth / wait time.
  • Errors — failed operations.

Cardinality discipline

  • No unbounded tags (user_id, tenant_id, url with params). They explode your TSDB bill.
  • Pre-aggregate cardinal dimensions; keep labels coarse.
  • http.route (the template), not http.url (the instance).

Correlation

  • request_id in: logs, traces (as request.id attr), response headers, error payloads.
  • trace_id in: logs (auto-injected by SDK), response headers for user bug reports.
  • A user reporting "I got a 500" should give you enough to find the request — the request_id is visible and searchable.

Per-language

Node

  • Pino or Winston with JSON transport.
  • OpenTelemetry SDK (@opentelemetry/sdk-node).
  • Prom-client for metrics.

Python

  • structlog with ProcessorFormatter.
  • opentelemetry-sdk + opentelemetry-instrumentation-*.
  • prometheus_client.

Go

  • slog (stdlib, Go 1.21+) with JSON handler.
  • go.opentelemetry.io/otel + auto-instrumentation.
  • prometheus/client_golang.

Rust

  • tracing + tracing-subscriber with JSON formatter.
  • opentelemetry-otlp.
  • prometheus crate.

Anti-patterns

  • "Log everything, we'll figure it out later" — the noise hides the signal you need.
  • Traces without sampling — you'll drown; sample intelligently (head-based at edges, tail-based for errors).
  • Metrics where logs would do — counting "user clicked button" 1M times/day is cheaper as a metric than a log.
  • Logs where metrics would do — "latency: 247ms" as a log line is wasting money; it's a histogram.
  • Alerts on everything — page only on symptoms that hurt users, not causes.

Smells

  • A single log line has more than ~10 fields — probably two events glued together.
  • You use grep to find things — you should have a field + dashboard.
  • Error messages are generic ("an error occurred") — they cost you the 2am minutes.
  • console.log in prod code — belongs in a logger, tagged and leveled.