experiment-execution

Execution runner for Orbit Wars experiments under `backend/pipeline/`. Takes a single hypothesis (one line in `hypotheses.md` or one `iterN_plan.md`) and drives the full cycle inline in the main sessi

SKILL.md

GitHub

Experiment Execution Skill (Orbit Wars)

Drives one hypothesis through plan → implementation → smoke → test → push → RunPod → monitoring → evaluation → iterN_result.md inline in the main session. Honors the hypotheses.md skip list to short-circuit phases (skip eval, skip replay, skip smoke, skip RunPod, skip auto-recover).

When this skill is in charge

  • The user typed /experiment-execution (explicit trigger), or
  • The user asked, in plain language, for any of:
    • launching a new case and training it
    • iterating an existing case and retraining
    • kicking off a RunPod training run
    • writing up the result of a recent run
  • The user did not already pre-specify the entire pipeline themselves (e.g. "just run dev/runpod train abc123" — at that point, just run it).

If the user is asking for read-only review, dead-code cleanup, Kaggle submission, or infra-only work, redirect to the matching skill/agent.

What "an experiment" means in this repo

  • Family — imitation / rulebase / reinforce.
  • Case — case<N> under backend/pipeline/<family>/.
  • Hypothesis — one sentence + adoption threshold.
  • Compute target — local CPU only / RunPod GPU.
  • Skip list — read from hypotheses.md > 実施しない検証 / 評価 and used to short-circuit phases.

Comply with .claude/rules/docs.md and .claude/rules/backend/pipeline.md.

Skill flow

8 phases. Phase 4.5 (smoke test) is mandatory. Phase 6 specifies which dev/runpod tail source to use. Phase 6.5 makes pod terminate explicit on auto-recover.

Phase 1 — Resolve the input source

Resolution priority:

  1. Explicit iterN_plan.md — if a plan already exists, hypothesis / scope / metric / skip list are read from the plan; the skill only does minimal confirmation.
  2. hypotheses.md H{n} reference — e.g. "H3 を実行して" → pull the line from hypotheses.md, internally call experiment-plan to write iterN_plan.md, then continue to Phase 2.
  3. Free-text hypothesis — internally write iterN_plan.md first, then continue.

Items to confirm (extract from iterN_plan.md / hypotheses.md; ask 1–2 questions only if missing):

  1. family / case
  2. one-sentence hypothesis + scope
  3. primary metric + default episode count (when 300 対戦 skip is set, mark 対戦評価 skip)
  4. compute target (RunPod GPU / local CPU only)
  5. skip list (smoke skip / dev/test-backend skip / replay 分析 skip / RunPod 不使用 / auto-recover 不使用)

Phase 2 — Reality check

Parallel Bash:

git status --short
git rev-parse --abbrev-ref HEAD
git rev-parse HEAD
ls backend/pipeline/<family>/
ls docs/experiment/<family>/

If the tree is dirty and a RunPod launch is planned, decide with the user (commit / stash / proceed without RunPod).

Phase 3 — Confirm or write iterN_plan.md

If Phase 1 found no iterN_plan.md, run an experiment-plan-equivalent flow internally (skip list honored to auto-shorten ## 検証方法). Otherwise reuse the existing plan.

If the user scoped the request to "plan only", stop here.

Phase 4 — Implement the case

Comply with .claude/rules/backend/pipeline.md:

  • pipeline/<family>/case<N>/main.py exists, exposes agent(obs), uses sys.path.insert(0, str(Path.cwd()))
  • Intra-package imports are relative
  • New case → register in backend/src/dataset/selfplay/agents.py (AGENT_REGISTRY)
  • Add dev-only directories to backend/pipeline/.submitignore

For existing-case extension, prefer adding a new module (policy_v2.py, features_v3.py) — preserves a comparable baseline.

Phase 4.5 — Local smoke test (mandatory)

Always run unless skip list explicitly sets smoke test (1-episode self-play) を skip. When skipped, surface a one-line note before moving to Phase 5.

In order:

  1. Import sanity:
    cd backend && uv run python -c "from pipeline.<family>.case<N>.baseline.agent import agent; print(agent)"
    
  2. 1-episode self-play smoke (new):
    cd backend && uv run python -m dataset.selfplay.run --case <family>/case<N> --episodes 1 --seed 0
    
    Or call the case's smoke harness. Failure here means no remote launch. This 1-episode pass is the last line of defense before paying for GPU time.
  3. dev/test-backend (mandatory unless skip list disables it):
    dev/test-backend
    
  4. Submit-shape changes → dry-run validator:
    uv run --directory backend python -m submit submit <family>/case<N> --dry-run -m "dry-run"
    

On failure, optionally delegate to python-build-resolver. Always pass before progressing to RunPod.

Phase 5 — Launch RunPod GPU

If skip list sets RunPod を使わない (local CPU only), skip Phases 5 / 6 / 6.5 entirely and go to Phase 7 (local evaluation).

Otherwise:

git push origin <branch>
dev/runpod train <commit-sha> [--case <caseN>] [--cloud-type SECURE|COMMUNITY|ALL]

Pre-flight:

  • git status clean
  • commit pushed

Save run_id / commit SHA / case / start time. Report 3 lines after launch and immediately set up the Phase 6.5 cron monitor.

Phase 6 — Inline reactive monitoring during the run

Respond to user requests with the right command:

user says command what you get
"今どこ?" / "進捗教えて" dev/runpod status <run_id> pod state / S3 progress marker / latest metrics
"train ログ見せて" dev/runpod tail <run_id> --source train (live) or dev/runpod logs <run_id> (post-mortem) training stdout (step / loss / lr / val_metric)
"setup ログ見せて" dev/runpod tail <run_id> --source onstart onstart.log (env setup / dvc pull / driver / mountpoint)
"GPU 使用率は?" dev/runpod tail <run_id> --source gpu nvidia-smi 10s sample (utilization / VRAM / temp)
"成果物今ある?" dev/runpod pull <run_id> DVC, falling back to S3
"止めて" confirm, then dev/runpod stop <run_id> (surface forfeited cost in 1 line)
"lr も変えたい" treat as a new iter; do not touch the in-flight run; queue iterN+1_plan.md —

NEVER claim "succeeded" until artifacts (best.pt, metrics JSON) are pulled and inspected.

For asynchronous notification on completion, use dev/runpod watch <run_id> (desktop notification) or dev/runpod train --watch at launch time.

Phase 6.5 — Cron-driven periodic health check & auto-recover

If skip list sets auto-recover loop を使わない, register the cron in monitor-only mode (notify, no stop / fix / relaunch).

Otherwise (default):

  1. Use the schedule skill (cron) at 10-min cadence (extend to 15–20 min for very long ETAs).
  2. Each tick runs dev/runpod status <run_id> (+ dev/runpod logs <run_id> --tail 20 when unhealthy) and applies:
state action
RUNNING + metrics advancing no-op
RUNNING + metrics stalled for 2 consecutive checks soft warning to user; no auto-stop
EXITED + success marker (best.pt + metrics JSON in S3) cancel cron → Phase 7
EXITED + failure (non-zero / OOM / crash) trigger auto-recover loop
Cost cap exceeded stop pod → surface to user immediately (no auto-relaunch)

Auto-recover loop (failure path):

  1. Stop & terminate the pod — dev/runpod stop <run_id>. Then terminate / detach volume to ensure the pod stops billing. Always perform an explicit terminate even if the pod looks crashed.
  2. Pull artifacts & logs: dev/runpod pull <run_id> --from s3 + dev/runpod logs <run_id>
  3. Diagnose: env/setup error (first-pass against memories project_runpod_onstart_pitfalls and project_runpod_5_traps_2026_05_04) / code bug / config / OOM / experiment code bug
  4. Fix in the worktree:
    • Code/config bug → edit, dev/test-backend, commit + push
    • OOM → batch size / GPU class change
    • Onstart / env → fix dev/runpod script or case-level setup
    • Infra flake (RunPod outage / GPU node lottery) → surface to user and stop (no auto-relaunch)
  5. Re-launch with the new SHA: dev/runpod train <new-sha> --case <caseN>.
  6. Re-arm cron for the new run_id, continue Phase 6.5.

Each step emits a Japanese status message ≤ 3 lines. When Phase 7 starts or the user terminates, cancel the cron and confirm via dev/runpod ps that no stale pod remains.

Phase 7 — Evaluation and iterN_result.md

Honor the skip list to determine evaluation scope:

  • ローカル self-play 300 対戦を行わない ON → judge from training logs only (loss curve / val_metric). Replace ## Numbers with a training-log table.
  • 100 対戦のみ ON → run 100-episode self-play; if n<300 で結論を出さない is also ON, fix decision = inconclusive.
  • default (300 ep) → standard evaluation (≥300 ep).

In order:

  1. dev/runpod pull <run_id> to fetch artifacts to data/output/models/<family>/case<N>/runs/<run_id>/.
  2. Run the case's evaluation script (backend/pipeline/<family>/case<N>/evaluation/compare_*.py or similar) — skip when not applicable.
  3. Decision is based on local match outcomes only (no Kaggle publicScore / skill rating).
  4. Write iterN_result.md next to plan.md:
# {Family}/case{N} — {Topic} (iter{N}) RESULT

> 関連: iterN_plan.md / hypotheses.md
> run_id: {run_id} / commit: {sha} / case: {caseN}
> 開始: {start} / 終了: {end} / コスト: ${cost}

## Summary
{1 段落: 仮説は支持されたか?}

## Numbers
{skip list に従い、対戦評価 table or 学習ログ table}

| metric | value | note |
|---|---|---|

## Diagnosis
{なぜ機能した / しなかったか}

## Decision
- 採否: {adopted | rejected | inconclusive}
- 次の一手: {follow-up 仮説 or `dev/runpod promote <run_id>` (user 確認必須)}

## Artifacts
- model: `data/output/models/<family>/case<N>/runs/<run_id>/best.pt`
- metrics: ...
  1. Report to the user in 3–5 lines: hypothesis result / key metrics / decision / result.md path.

Phase 8 — Provisional write-back to hypotheses.md

Even if experiment-analysis will run later, append exactly one row to the hypotheses.md Iteration log when iterN_result.md is finalized:

| {N} | {start} | H{n} | iterN_plan.md | {run_id} | {primary metric} | {adopted/rejected/inconclusive} | iterN_result.md | (analysis 未実施) |

Do not flip the hypothesis-row checkbox here (that is experiment-analysis Phase 4.5's job). Phase 8 only adds the Iteration log row.

Skip Phase 8 entirely when hypotheses.md does not exist (one-shot invocation).

Kaggle submission is out of scope (per .claude/rules/command.md; explicit approval + dev/submit).

Risk gates this skill enforces

  • Cost-cap overruns auto-stop. Phase 6.5 stops the pod and never auto-relaunches when the cost cap is exceeded. Recovery from crash / OOM / bug is fine; budget overruns require user approval.
  • Smoke test (1-ep self-play) is mandatory by default. Honor skip list only when the user explicitly disabled it. This is the last line of defense before remote training spend.
  • Auto-recover always terminates the pod. Stop alone is insufficient; ensure the pod is terminated and the volume detached so charges stop.
  • Cancel the Phase 6.5 cron when Phase 7 starts or the user terminates the run.
  • Honor the hypotheses.md skip list (300 対戦 / replay / smoke / RunPod / auto-recover flags all map to phase short-circuits).
  • Hypothesis-row checkboxes belong to experiment-analysis. This skill only appends an Iteration log row.
  • n<300 verdicts are fixed at inconclusive. Per memory project_imitation_case1_phase3.
  • Never use Kaggle publicScore / skill rating as evidence.
  • Don't launch RunPod from a dirty tree.
  • Canonical-weights update (dev/runpod promote) and Kaggle submission are out of scope. User must approve those individually.

Common shapes

User says… Skill behavior
"imitation/case1 で dropout を 0.3 にして回して" Phase 1 (1 question to confirm full RunPod run) → Phases 2–8 inline.
/experiment-execution H3 Phase 1 pulls H3 from hypotheses.md → internally write the plan → Phases 2–8.
"case4 の plan.md だけ先に書いて" Phases 1–3 only; skip Phases 4 onwards.
"さっきの run xxx の結果を docs にまとめて" Phase 1 + 7 + 8 only (pull + eval + result.md + Iteration log row).
"今 RunPod で回ってる run は終わった?" Phase 6 inline status.
"やっぱり lr も変えたい" (mid-flight) Treat as a new iter; do not touch the in-flight run; queue iterN+1_plan.md.
"smoke 飛ばして直接 RunPod" Check skip list; if absent, ask one confirmation; only then proceed without smoke.

Things to avoid

  • Spawning a subagent for the long-running portion (Phases 5–6). Inline is required so the user can interject.
  • Forgetting to cancel the Phase 6.5 cron at Phase 7 / on terminate.
  • Auto-relaunching after a cost-overrun stop. Crashes / bugs / OOM are recoverable; budget overruns need explicit user approval.
  • Polling more frequently than every 5 minutes (cache waste, no signal).
  • Citing Kaggle publicScore as the success metric.
  • Creating a new experiment directory when the user is iterating (use iterN_*.md instead).
  • Putting machine-generated artifacts under docs/experiment/ (those go to data/output/experiment/).
  • Claiming a run finished before pulling and inspecting artifacts.
  • Flipping the hypothesis-row checkbox in hypotheses.md (Phase 8 only adds Iteration log rows).
  • Running the default eval flow without checking the skip list (e.g. running 300 episodes when 300 対戦 skip is set wastes GPU).
  • Stopping a pod without terminating it. Attached volume = ongoing charges.

Communication cadence

  • One short message at each phase boundary (plan written / case implemented / smoke green / tests green / RunPod launched / run finished / result written). Each ≤ 3 lines.
  • After RunPod launch, surface run_id + ETA, then return control. Do not narrate while waiting.
  • Status checks: ≤ 3 lines (status / latest metric / ETA delta).
  • End each turn with what's next, e.g. "run_id=abc123, ETA ~45min — status で進捗、止めて で中断、結果まとめて で完了後の集計に進みます".

Language

  • Internal reasoning and thinking should be in English
  • All user-facing output, AskUserQuestion labels/descriptions, and result reports must be in Japanese (per .claude/CLAUDE.md)