diagnose-stuck-or-failed-run
Use when classifying a stuck, failed, empty, or zero-scoring evaluation as infrastructure or policy failure.
SKILL.mdtemplates/optimizers/242026/crewrift-specific/skills/diagnose-stuck-or-failed-run/SKILL.md
GitHubDiagnose a stuck or failed eval run (infra vs policy)
The reflex when a run looks dead is to blame the policy and revert, or to assume a crash and resubmit. Both are usually wrong. The vast majority of "dead run" signals are infra, not the policy. This recipe classifies the failure first.
Golden rule (evidence discipline): only a TERMINAL, completed round with real per-player scores is policy evidence. Pending / submitted / claimed / running rounds, empty logs, and rounds that fail before a player process starts are non-evidence. Never replace a champion or revert a policy from non-terminal or failed-before-startup rounds alone.
Step 0 — Identify which symptom you have
Match your situation to a branch, then run that branch's checks:
| Symptom | Branch |
|---|---|
| Detached/background eval logfile is empty or stuck; "is it dead?" | A — Empty/buffered log |
| A submit printed a JSON decode error or "failed" — about to resubmit | B — uv preamble corrupts JSON |
Round stuck pending / running forever; hosted dispatch not picking up |
C — Backend / dispatch |
Empty division leaderboard, or coworld results crashed |
D — Empty leaderboard trap |
Round reached terminal failed, or completed with zero/garbage scores |
E — Failed/zero terminal round |
A 422 on submit, or fields the server rejects |
F — API shape drift |
Don't skip ahead. Each branch ends with a classification (INFRA or POLICY) and a success check.
Branch A — Empty / buffered detached log ("is my run dead?")
A long-running Python eval launched detached (nohup, background, output redirected
to a file) block-buffers stdout: the logfile stays empty until the process exits,
even while the run is healthy and working. An empty log is NOT failure evidence.
Before declaring it crashed, check liveness and side-effects (in order, cheapest first):
Process is alive:
ps -p <PID>Alive → it is buffering, not dead. Stop here unless other signals contradict.
Side-effect artifacts exist: the eval writes a run directory as soon as the submit succeeds and the poll loop starts. A created run dir = submit succeeded and the loop is running. Check the run-dir tree (per-xreq folder with
episode_NN/subfolders + asummary.jsonfor theantfarm_run.pyartifact path).Live docker container count (local docker path only):
docker ps -q | wc -lA non-zero, structured count means games are running. (Session-derived example: 54 containers = 6 concurrent games × 9 containers each.) (session-derived, unverified)
Query the server for the request's status directly rather than trusting the log — poll the experience-request / round status via the API (see Branch C for the status check). The server is the source of truth, not the buffered file.
The fix (so this never recurs): force unbuffered stdout in the eval/diagnostic
script — PYTHONUNBUFFERED=1, or python -u, or
sys.stdout.reconfigure(line_buffering=True). Apply this even to throwaway diagnostic
scripts; the same mistake recurs there. (session-derived, unverified)
Long-running watches: run a continuous watch loop via the Monitor tool with an
until-loop, NOT a detached Bash shell with a long foreground sleep — the detached
shell gets reaped mid-sleep and the watch dies silently. The monitor.py tool (the
standalone league rank/round watcher) is the reference pattern for "always-on,
print-only-on-change". (session-derived, unverified)
Classification: process alive OR artifacts present OR server says running → INFRA (buffering), run is healthy. Do not resubmit, do not revert. Wait for terminal status or query the server.
Branch B — A submit "failed" with a JSON decode error (uv preamble)
uv run writes an install/sync preamble to stdout, which intermittently corrupts
machine-parsed JSON output. A CLI whose JSON you pipe into a parser will throw a JSON
decode error even though the underlying command SUCCEEDED — only the parse failed.
Do this:
Do NOT resubmit on a parse error. The submit may have created the request. Resubmitting double-submits. (Session-derived: three consecutive "failed" submits had all actually been created.) (session-derived, unverified)
Verify server/queue state before any resubmit — query the experience-request / round list and confirm whether your request already exists (see Branch C status check). Only resubmit if the server has no record.
Fix the invocation: when a CLI's JSON must be parsed, invoke the project's venv python directly instead of through
uv run:.venv/bin/python <script> ...This skips the
uvpreamble so stdout is clean JSON.
Classification: parse error with a created request on the server → INFRA (uv preamble), submit succeeded. Not a policy or backend failure.
Branch C — Round stuck pending / running / dispatch not picking up
First, decide the execution backend. For crewrift experience requests,
execution_backend="k8s" is the dependable hosted path. antfarm is slow-dispatch and
unreliable — measured ~61% episode failure rate, a ~60s/episode cold-start floor, and
multi-hour outages; a matched probe had k8s complete 4/4 in 239s while antfarm sat
~243s pending then failed all 4 on 502/read-timeout. If you have a choice, use k8s.
The A/B eval engine (eval.py → league_eval.run_league_ab / run_arm) already
encodes a formula-derived timeout budget and a wedge detector — let it run rather than
declaring failure early:
- Total arm budget:
~ num * maxticks/24 * 3 + 900seconds. Games run at realtime sim pacing (TargetFps 24), so a fixed short timeout falsely fails slow-but-healthy episodes. One episode atmaxTicks 10000÷ 24 ≈ 7 minutes wall time; episodes parallelize fully server-side, so the budget is dominated by the longest single episode + warm-up, notnum× episode-time. - Wedged-participant detection: if ZERO episodes complete within
maxticks/24*3 + 600seconds, a participant is wedged — fail LOUD.run_armdoes this withstall_grace_seconds(transient statuses404/425/429/5xxare retried; zero completions past the grace = declared wedged). Poll cadence ~10s.
Status-probe a stuck round / request directly:
- Poll the experience-request to terminal via
GET/list on/v2/experience-requests(the parent) and its child episodes. A status ofsubmitted/claimed/runningis non-evidence — keep waiting up to the budget above. - If on antfarm and "stuck running": the episode may have already finished. Hit the
antfarm episode worker's own
/healthzand/results.status=finished+404results = the game completed but emitted no artifacts (a game-side gap, not slowness). (session-derived, unverified) - antfarm "pending forever" / registration: antfarm rejects any roster containing a
policy version with no antfarm registration row ("Policy version ... is not registered
with Antfarm"), and top-N/league selection is NOT registration-aware, so it picks
unregistered policies and fails at dispatch. k8s has no such requirement — the
fastest fix is to re-run the same roster on k8s. (Registration names embed
sha256(policy_version_id)[:12]as a suffix; if you must stay on antfarm, intersect the registrar's public/listwith live league memberships to compute the dispatchable set, then pin all seats explicitly.) (session-derived, unverified) - Routing quirk: prod Observatory dispatches antfarm jobs to antfarm staging
workers; use prod (not a local backend) for antfarm requests. A completed run with
live_url=Noneran on k8s, not antfarm (only antfarm dispatches carry alive_url) — don't count a k8s success as antfarm proof. (session-derived, unverified) - Disambiguate an outage with controls: run a known-good simple game on antfarm (tells antfarm-wide vs game-specific) and run the same roster on k8s (tells antfarm-specific vs systemic).
Classification:
- Within budget, status pending/running → INFRA (still working), wait.
- Past wedge grace with zero completions → INFRA (wedged participant / backend), fail loud and re-run on k8s. Not policy.
- antfarm down but k8s completes the same roster → INFRA (antfarm). Switch to k8s.
Branch D — Empty leaderboard / coworld results crash
An empty division leaderboard does not mean an empty division. The leaderboard only
shows SCORED champions, so 0 rows can coexist with a populated division (observed: 0
leaderboard rows alongside member_count=10; a champion-eligibility pool of 5 while the
broader competing-membership pool held 136). This is the trap that makes server-side
top_n champion auto-fill silently fail to fill a 7-seat field even though plenty of
runnable opponents exist.
Do this:
- Do NOT infer "no eligible opponents" from an empty leaderboard. Check membership
counts and the live-membership pool separately — the
softmax_pull.pytool already surfaces division leaderboards, rounds, and submissions from the endpoints that work today (and coerces a null leaderboard to[]), so use itsfetch_softmax(league_id)path for the outcome side rather than reading the leaderboard alone. coworld results <division_id>(0.1.16) CRASHES on an empty leaderboard: the API returns null and the client validates againstlist[LeaderboardEntryPublic](pydantic "Input should be a valid list"). Divisions with entries work fine. Workaround:coworld memberships --mineto check a qualifying bot's status. (Filed as Metta-AI/coworld issue #11.) (session-derived, unverified)- This is most common on a freshly-entered division (e.g. Qualifiers right after a submission), before any round has scored.
Classification: empty leaderboard with a non-empty membership pool → INFRA /
expected state, not a dead division. Don't dismiss top_n or conclude "no opponents".
Branch E — Failed / zero-score terminal round (the revert trap)
A hosted round that comes back failed or with zero/garbage scores is often a
runtime/infra failure, not a policy regression. Documented cases where reverting the
policy would have been wrong:
- Auth crash before any player starts (e.g.
KeyError: 'authorization'/ missing in-cluster auth). No player ran → the round is zero policy evidence. - Game-image route/version mismatch: the worker requested one client path while the
deployed game image served a different one → the round
404s. Packaging/infra skew, not a policy bug. - Recorder error like "results contain N scores for M stored policy versions" → points at round recording / runtime state; reproduces even on a fresh upstream reference bot, so it is not your policy.
- Teardown race: the episode finished and the server was ready to write
results.json, but a player process stayed blocked in websocket teardown and had to be force-killed. If the player's process timeout is too short, the player disconnects before the server records the round → no usable results even though game logic completed. (And the runner-side image gate's own timeout must be long enough that aresults.jsonarriving just after a stingy deadline isn't discarded as a runner timeout.)
The classification procedure:
- Confirm a player actually started and produced a score before blaming the policy.
Inspect the per-episode-request record, the downloaded artifacts
(
results.json+ game logs + per-agent policy logs — theantfarm_run.pyharvest path pulls all of these intoepisode_NN/folders), and any submission dry-run notes. - No player process started (auth/route/recorder error before startup) → INFRA, non-evidence. Do not revert.
- Player started but disconnected during teardown / no results despite completed game
logic → INFRA (teardown race). The fix is operational, not a policy revert:
wrap the player's run command in a GNU
timeoutwith a generous duration and--kill-after, and MAP the timeout exit code to success — a player reaped after the game is done is normal lifecycle, not a failure. Set the timeout comfortably longer than a full episode (tens of seconds disconnected players before results were written; minutes fixed it). This istry/finally-style lifecycle handling, not error-swallowing. - Only if a TERMINAL round with real per-player scores shows the regression, reproduced across enough rounds, is it POLICY evidence.
If the live policy genuinely is broken: fix it operationally — upload a clean replacement and let the new placement de-champion the old version — rather than adding a backward-compat shim for an obsolete import path.
Classification: failed/zero round with no started player or a teardown race → INFRA, non-evidence. Real per-player scores reproducing a regression → POLICY.
Branch F — 422 on submit / API shape drift
Conform to the deployed server's OpenAPI, not your local client models — the request
shape drifts. POST /v2/experience-requests was reworked on 2026-06-12 to a unified
roster body (one entry per seat, policy_ref). The old requester/opponents /
policy_version_ids / top_n / target shape now 422s.
- The league A/B path (
eval.py→league_eval.build_request_body) is already on the post-rework shape: the seat is pinned inside therequesterobject (requester.slot) — the server rejects a top-levelrequester_slotsupplied next to arequester. antfarm_run.py(build_request) still emits the pre-rework shape and will 422 until patched to the unifiedrosterbody. Re-fetch live members and validate every pvid right before launch; the league rolls policy versions mid-session.
Classification: a 422 on submit → INFRA (API shape drift). Patch the request
body to the deployed schema; not a policy failure.
Success check (you are done when)
You can state, for the specific run, one classification with the evidence that backed it:
- INFRA — and you named the cause (buffering / uv preamble / antfarm down / empty
leaderboard / auth-before-startup / teardown race / API drift) and took the infra
action (wait, query server, switch to k8s, fix invocation, patch body, raise the player
timeout). You did not revert or replace the policy on this evidence. - POLICY — and you have a TERMINAL completed round (or A/B verdict) with real per-player scores that reproduces the regression across enough rounds to clear the verdict bar. Only then is reverting / re-uploading justified.
If you cannot produce a terminal round with real scores, the answer is INFRA-or-unknown, and the run is not policy evidence — do not touch the policy.