go-observability
Golang observability patterns — logs, metrics, and traces. Covers structured logging (slog), Prometheus metrics, and OpenTelemetry tracing.
SKILL.mdskills/go/go-observability/SKILL.md
GitHubThe Three Pillars
- Logs: What happened? Use structured logging with
log/slog. - Metrics: How much/how fast? Use Prometheus client for aggregated measurements.
- Traces: Where did time go? Use OpenTelemetry (OTel) for distributed tracing.
Best Practices
- Structured Logging (JSON) — Production services MUST emit structured logs (JSON), not freeform strings.
- Log with Context — Use
slog.InfoContext(ctx, ...)to correlate logs with traces. - Histogram over Summary — For latency metrics, prefer Histograms as they support server-side aggregation.
- Low Cardinality Labels — NEVER use unbounded values (user IDs, full URLs) as Prometheus label values.
- Propagate Context — Context is the vehicle that carries
trace_idandspan_idacross service boundaries. - Golden Signals — Always monitor Latency, Traffic, Errors, and Saturation.
- Single Handling Rule — Errors MUST be either logged OR returned, NEVER both.
Code Examples
Structured Logging (slog)
slog.Info("order processed",
"order_id", order.ID,
"user_id", user.ID,
"amount", order.Amount,
)
Prometheus Metrics
var (
httpRequests = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests.",
}, []string{"method", "route", "status"})
)
func middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
next.ServeHTTP(w, r)
httpRequests.WithLabelValues(r.Method, r.URL.Path, "200").Inc()
})
}
OpenTelemetry Tracing
func (s *Service) DoWork(ctx context.Context) error {
ctx, span := otel.Tracer("my-service").Start(ctx, "DoWork")
defer span.End()
// Perform work, span context is propagated via ctx
return s.repo.Save(ctx, data)
}
Anti-patterns
- ❌ High cardinality labels — e.g., using UserID in Prometheus labels (crashes Prometheus).
- ❌ Log and Return — Causes duplicate logs and noise.
- ❌ Not passing context — Breaks the trace chain and makes debugging across services impossible.
- ❌ Using
fmt.Printforlog.Printffor production logs (non-structured). - ❌ Missing error recording in spans — Use
span.RecordError(err)to see failures in your tracing tool.