jobs-trace-analyzer

Analyze benchmark jobs (Terminal-Bench, Harbor) in jobs/, inspect ATIF trajectory logs, tool call errors, prompt issues, timeout/circuit-breaker failures, and verifier results to identify harness bott

SKILL.md

GitHub

Jobs Trace & Harness Bottleneck Analyzer

Analyze benchmark run trajectories stored under jobs/<job_timestamp>/ to pinpoint harness bottlenecks in LibrAgent (Builtin Tools, Prompts, Circuit Breakers, Timeouts, Context Compaction) that degrade scores, and formulate evidence-backed improvement plans.

Scope: tool sequences, observations, harness layers. Don't care: model, serving engine. No harness contract broken → pattern and stop.

Relationship to harbor-harness-improvement-loop

This skill is a lighter inventory / categorization pass over jobs/<run>/.

Concern Use
Quick pass-rate, failure buckets, tool frequency jobs-trace-analyzer (this skill)
Evidence-backed BM → fix → rerun cycles harbor-harness-improvement-loop

Harbor’s analyzer is stricter: verifier/reward.txt as reward SSOT, ATIF-first metrics, and explicit heuristic ≠ verified failure. Prefer that skill when proposing product changes. Do not treat this script’s categories as causal proof.

Workflow

1. Job Inventory & Automatic Trace Analysis

python .agents/skills/jobs-trace-analyzer/scripts/analyze_jobs_trace.py \
  jobs/<job_timestamp> \
  --output .libragent/work/trace-analysis/<job_timestamp>-analysis.md

Optional: --verbose prints per-task tool name lists.

The script reports:

  • Total trials, pass rate (%), failed trial counts
  • Failure categories (see below)
  • Tool call frequency (ATIF-first when metadata trajectory is empty)
  • heuristic_error_observations (keyword hits in ATIF observation text — not verified tool failures)

Failure categories

Category Meaning
SUCCESS Reward ≥ 1.0
TIMEOUT_EXCEEDED Exception type contains Timeout
NETWORK_API_ERROR Connection / API network exception
TOOL_EXECUTION_ERROR Structured tool error fields only (rare; not observation heuristics)
INITIALIZATION_OR_EARLY_ABORT Zero turns
AGENT_LOOP_OR_STUCK ≥3 repeated adjacent identical tool calls (loop detected)
HIGH_TURN_COUNT Turns > 30 without repeated-call evidence (inspect manually)
VERIFIER_FAILED_WRONG_STATE Default failed bucket

Tool extraction priority

  1. agent_result.metadata.trajectory tool calls when present
  2. Else ATIF agent/trajectory.json → steps[].tool_calls[].function_name
  3. Else metadata.tool_calls_count (count only; names unavailable)

Reward prefers verifier/reward.txt, then verifier_result.rewards.reward.


2. Deep-Dive Trajectory Inspection (ATIF & Task Logs)

For each failed trial, inspect jobs/<job_timestamp>/<task_name>/:

  1. result.json: rewards, exception_info / exception.txt, metadata turns
  2. agent/trajectory.json (ATIF): first divergence turn
  3. verifier/reward.txt & trial.log: why verification failed

3. Harness Layer Mapping

Don't care (never diagnose, compare, or blame): model, serving engine. In this harness project, tests evaluate the harness itself. If no harness contract (tool schema, handler, prompt assembly, circuit breaker, adapter) is broken, report the tool pattern and stop.

Symptom Primary Owning Layer Target Code
Tool parameter / schema error Builtin MCP tools src-tauri/src/mcp/builtin/
Repeated tool loop Circuit breaker / natural recovery src-tauri/src/agent/llm/circuit_breaker.rs, natural_recovery.rs
Timeout / unfinished Timeout multipliers / shell src-tauri/src/mcp/builtin/workspace/utils.rs
Missing verification / premature end Prompts src-tauri/src/agent/prompt/
Context loss / truncation Compaction src-tauri/src/agent/llm/completion/request/compact.rs

4. Formulate Evidence-Based Improvement Plan

For each issue: symptom + trajectory evidence → root cause → smallest intervention → verification (pnpm refactor:validate, cargo test, re-run slice). Escalate to harbor-harness-improvement-loop for controlled BM cycles.


5. Report Output Location

Save reports under:

.libragent/work/trace-analysis/<job_name>-analysis.md