observability
Add structured logging, metrics, and distributed tracing to any service or monolith
SKILL.mdskills/cross-cutting/observability/SKILL.md
GitHubObservability
Context
Observability is the ability to understand the internal state of a system from its external outputs. The three pillars are logs, metrics, and traces. A system without observability is a black box — you cannot debug production issues, validate deployments, or understand user-facing behavior.
Works in monoliths (single service, shared log/metrics output) and distributed systems (requires correlation IDs and distributed tracing).
Use this skill when:
- Instrumenting a new service or feature
- Diagnosing a production issue with poor visibility
- Reviewing observability coverage before a major release
- Setting up alerting for a system that lacks it
Instructions
Pillar 1 — Structured Logging
Rules:
- Log in structured format (JSON). Never use unstructured string concatenation. Structured logs are queryable.
- Every log line must have: timestamp, log level, service name, correlation/trace ID, and message.
- Log at the right level:
ERROR— unexpected failure requiring investigationWARN— unexpected but handled, or degraded behaviorINFO— significant business events (order placed, user logged in)DEBUG— developer diagnostics (do not enable in production by default)
- Never log sensitive data: passwords, tokens, PII, payment data.
- Include context, not just messages. Bad:
"payment failed". Good:{"event": "payment_failed", "order_id": "abc", "reason": "gateway_timeout", "attempt": 2}.
Minimum fields on every log entry:
{
"timestamp": "2025-03-15T21:00:00Z",
"level": "ERROR",
"service": "checkout-service",
"trace_id": "abc123",
"span_id": "def456",
"message": "Payment gateway timeout",
"order_id": "ord_789",
"duration_ms": 5023
}
Pillar 2 — Metrics
Key metric types:
| Type | Use for | Example |
|---|---|---|
| Counter | Things that accumulate | http_requests_total |
| Gauge | Current values | active_connections, queue_depth |
| Histogram | Latency and size distributions | http_request_duration_seconds |
The Four Golden Signals (Google SRE): Instrument these for every service:
- Latency — time to serve a request (separate success vs. error latency)
- Traffic — requests per second
- Errors — error rate (4xx, 5xx, application errors)
- Saturation — how "full" the service is (CPU, memory, queue depth)
USE Method (for infrastructure):
- Utilization — % time resource is busy
- Saturation — amount of queued/waiting work
- Errors — error count
Pillar 3 — Distributed Tracing
For monoliths: Add trace IDs to log lines to correlate events within a single request lifecycle.
For distributed systems:
- Generate a
trace_idat the entry point (API gateway or first service to receive the request). - Propagate
trace_idandspan_idin HTTP headers (traceparentin W3C Trace Context format). - Every service creates a child span with its own
span_idand the parent's ID. - Instrument slow or critical operations as explicit spans.
OpenTelemetry is the standard instrumentation library — use it over vendor-specific SDKs where possible.
Alerting Principles
- Alert on symptoms, not causes. Alert when users are affected (high error rate, high latency), not when an internal metric crosses a threshold.
- Every alert must be actionable. If you don't know what to do when an alert fires, it should not be an alert.
- Link every alert to a runbook. The alert body should include a direct link to the relevant runbook.
- Avoid alert fatigue. A team that ignores alerts is more dangerous than a team with no alerts.
Observability Readiness Checklist
- All services emit structured JSON logs
- Logs include trace ID and service name
- Four golden signals are instrumented and dashboarded
- Alerts exist for error rate and latency SLOs
- Distributed traces propagate across all service calls
- No sensitive data in logs
- Runbooks exist for all active alerts
Principles
- Source: Observability — Martin Fowler
- Source: Google SRE Book — Four Golden Signals
- Key idea: "Observability is not about collecting more data. It is about having the right data to answer arbitrary questions about your system's behavior in production without deploying new code."
Output Format
When applying this skill, the agent should:
- Audit existing observability coverage against the checklist
- Add structured log statements at key entry/exit points
- Instrument the four golden signals for any new service
- Generate the trace propagation boilerplate for the relevant language/framework
- Produce a prioritized list of observability gaps to address