instrument-llm-observability
Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing",
SKILL.mdskills/instrument-llm-observability/SKILL.md
GitHubInstrument an LLM app for observability
Add production-grade tracing to an LLM/agent app so every run is inspectable: prompts, tool calls, retrievals, token usage, latency, and cost per span.
Decide the approach first (2 questions)
- Is a backend already chosen? (Langfuse, Arize Phoenix, Comet Opik, Datadog, Grafana…). If yes, use its SDK/auto-instrumentation. If no, default to OpenTelemetry GenAI semantic conventions so traces export to any OTel backend and you avoid lock-in.
- Lowest-friction path available? If the app calls LLMs through a gateway/proxy (LiteLLM, Helicone, Portkey), you may get tracing + cost with zero code by changing the base URL. Check that before adding an SDK.
Vendor-neutral path (recommended default): OpenTelemetry
Emit gen_ai.* spans via an OTel instrumentation library, then point at any collector.
- Use OpenLLMetry (
traceloop-sdk) or OpenInference (Arize) - both emit OTel-compatible LLM spans. - Minimal shape:
- Install the instrumentation lib + an OTLP exporter.
- Initialize once at startup (set service name + OTLP endpoint via
OTEL_EXPORTER_OTLP_ENDPOINT). - Auto-instrument the LLM SDK (OpenAI/Anthropic/etc.); wrap custom agent steps in manual spans (
gen_ai.operation.name, tool spans).
- Result: spans carry model, prompt/completion, token counts, latency; cost is derived downstream.
See references/opentelemetry.md for the exact span attributes to set and a copy-paste init.
Platform SDK path (if a backend is already picked)
Wrap the LLM client with the platform's tracer (e.g. track_openai(...), @observe, an OTel exporter to that backend). Prefer their auto-instrumentation over hand-rolled spans; add manual spans only for custom tool/retrieval steps the SDK can't see.
Instrument the right things (checklist)
- Each LLM call: model, input messages, output,
prompt_tokens/completion_tokens, latency. - Tool/function calls: name, args, result, error.
- Retrieval steps (RAG): query, retrieved doc IDs/scores.
- Agent/chain boundaries: one parent span per user request, child spans per step.
- Errors: record exceptions on the span (don't swallow them - a silent
exceptthat returns empty output hides the failure). - Cost: derive from token counts × model price (most platforms do this for you).
Verify
Run one representative request and confirm the trace tree shows the full chain (parent → LLM/tool/retrieval children) with tokens + latency populated. If spans are missing, the instrumentation isn't wrapping that code path - instrument it manually.
Anti-patterns
- Logging raw prints instead of structured spans (no trace tree, no cost rollup).
- Instrumenting only the top-level call, missing tool/retrieval steps (the interesting failures hide there).
- Hard-coding a vendor SDK when OTel would keep you portable.
- Sending full prompts/PII to a third-party backend without redaction - mask sensitive fields first.
Tool picker
Not sure which backend? See the sibling choose-observability-stack skill, or the curated list in this repo's README.