langfuse
Look up ConvoPilot traces in Langfuse by run_id — debug test runs, inspect turn-by-turn spans, and identify evaluation failures
SKILL.md.claude/skills/langfuse/SKILL.md
GitHubYou are debugging a ConvoPilot test run using Langfuse traces.
Run ID: $ARGUMENTS
Langfuse Environment
Read from environment variables:
LANGFUSE_PUBLIC_KEY — Langfuse project public key
LANGFUSE_SECRET_KEY — Langfuse project secret key
LANGFUSE_HOST — Langfuse host URL (e.g. https://cloud.langfuse.com)
If env vars are not set, inform the user: "Langfuse is not configured — set LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST in .env." Do not proceed with API calls.
ConvoPilot Trace Hierarchy
TestRun (root trace — session_id = run_id)
└── TestCase (span)
└── ConversationTurn (span — one per turn)
└── ToolCall (event — one per tool call)
Key fields to look for:
| Span | What it tells you |
|---|---|
TestRun |
Overall status, total turns, run duration |
ConversationTurn |
role (user/agent), message, latency_ms |
ToolCall |
tool_name, arguments, output, validation_status (pass/fail/missing) |
EvaluationResult |
verdict, confidence_score, metric_results |
Step 1 — Fetch the session
AUTH=$(echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64)
HOST="${LANGFUSE_HOST:-https://cloud.langfuse.com}"
RUN_ID="<run_id from arguments>"
# Get session (contains list of traces)
curl -s "$HOST/api/public/sessions/$RUN_ID" \
-H "Authorization: Basic $AUTH"
Step 2 — Fetch observations for each trace
# Get all spans for a trace
curl -s "$HOST/api/public/observations?traceId=$TRACE_ID&limit=100" \
-H "Authorization: Basic $AUTH"
# Get only evaluation generations (LLM judge calls)
curl -s "$HOST/api/public/observations?traceId=$TRACE_ID&type=GENERATION&limit=20" \
-H "Authorization: Basic $AUTH"
Step 3 — What to look for
- Failed turns: Any
ConversationTurnspan withlevel=ERROR - Slow LLM calls: Evaluation generation spans > 3s latency
- Tool call failures:
ToolCallevents withvalidation_status=failorvalidation_status=missing - Low confidence:
EvaluationResultspan withconfidence_score < 0.7→ triggers human review - Run stuck in
running: Check if agent returned 5xx — look forstatusMessageonTestRunspan - Evaluation inconsistency: Check
raw_judge_response— if contradicts the verdict,temperaturemay not be 0
Step 4 — Present findings
Show the user:
- Run status: pass / partial / fail / stuck
- Timeline: chronological list of turns with role + latency
- Issues found: any errors, slow spans, failed tool calls
- Confidence score and which metrics dragged it down
- Direct Langfuse link:
$LANGFUSE_HOST/project/<project_id>/sessions/<run_id>
Error handling
| Error | Action |
|---|---|
| Session returns empty | Check that LANGFUSE_HOST points at the Langfuse instance/region the backend writes to |
| Auth fails (401) | Check that LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set correctly |
| No spans found | Run may still be running — check GET /simulation/runs/{run_id} on the backend API |
confidence_score missing |
Evaluation hasn't run yet — trigger GET /evaluation/{run_id} |