autoresearch
Bounded improvement loop: one artifact, fixed eval score, keep an edit only if strictly better. Use for AR_DESIGN/AR_FUNCTION/AR_ACCEPTANCE stages or standalone bounded iteration.
SKILL.mdskills/autoresearch/SKILL.md
GitHub<!-- desc:full -->
Full description
Shared bounded-loop runner for the Karpathy autoresearch pattern applied to code: ONE mutable artifact + a FIXED evaluation harness emitting ONE comparable composite score, looped as snapshot -> one focused edit -> score -> keep-if-strictly-better-else-revert -> log, bounded by per-pass max_iterations / max_consecutive_reverts and the cost brake. Invoked by the pipeline-orchestrator for the AR_DESIGN / AR_FUNCTION / AR_ACCEPTANCE stages, or runnable standalone. Validation only; NEVER auto-commits and NEVER runs destructive git.
Autoresearch — bounded keep/revert runner
This skill is the runner. It does not define what to optimize or how to score —
those live in the three per-pass prompt files. The runner owns the loop spine:
setup, one-edit-at-a-time iteration, snapshot/keep/revert, the results.tsv log, and the
bounded stop conditions. It is adapted directly from the AMS PRD autoresearch
program.md, generalized from "edit one PRD to maximize a reflection score" to "edit the
production source touched by a task to maximize a per-pass composite score."
Invariant (Karpathy autoresearch): ONE mutable artifact + a FIXED harness emitting ONE comparable score, then
loop { pick one idea -> edit -> score -> keep-if-better-else-revert -> log } -> stop on threshold / plateau / cap. You read this file and follow it.
How it is invoked
This runner has three modes, one per pipeline pass. The active mode selects exactly one per-pass prompt file, which is the authoritative source for that pass's mutable surface, fixed harness, rubric/KPIs, composite formula, and hard constraints:
| Mode | Pipeline stage | Per-pass prompt file (READ THIS for the active mode) |
|---|---|---|
design |
AR_DESIGN (AR-1, pre-TDD) | program.ar-design.md |
function |
AR_FUNCTION (AR-2, post-TDD) | program.ar-function.md |
acceptance |
AR_ACCEPTANCE (AR-3, post-QA) | program.ar-acceptance.md |
Two entry paths:
Pipeline-orchestrated (normal). The
/pipelinestate machine invokes this runner atAR_DESIGN(after CODE_REVIEW),AR_FUNCTION(after TDD), andAR_ACCEPTANCE(after PHASE_GATE). The orchestrator passes the mode and fills all placeholders from.aid/context.json+.aid/pipeline/config.json. On stop the runner FIRST restores the editable set to the best-kept version, THEN logs final scores tostep_historyand writes.aid/pipeline/state.jsonunconditionally as the FINAL action — after the last source mutation of ANY kind (kept edit, revert/restorecp, or crash restore), so the Stop hook (dev-pipeline-gate.sh) freshness check still passes (newest source mtime <= state mtime).Standalone. A human runs it directly on a single mode. Resolve placeholders yourself from
.aid/context.json+.aid/pipeline/config.json; if a value is absent, use the defaults noted in the per-pass file and state the assumption in chat.
Order matters and is enforced upstream:
functionmode requires a green test suite to exist (TDD produced it);acceptancemode requires QA to have run. Do not invent a harness a pass does not yet have — that is precisely why the passes are split.
Placeholders (filled at runtime)
Resolved from .aid/context.json and .aid/pipeline/config.json. The per-pass file declares
which it uses; the runner only needs these for the loop spine:
| Placeholder | Source | Meaning |
|---|---|---|
{{WORKSPACE}} |
context.json | Repo root path |
{{TASK_ID}} |
context.json | Current task id; names the results dir |
{{CRITICALITY_PROFILE}} |
context.json / config.json | Risk profile gating ASK-before-edit behavior |
{{KPI_TARGET}} |
config.autoresearch.kpi_target (default 9.5) | Quality-score target (0–10) |
{{TEST_COMMAND}} |
config.test_commands.unit (+coverage) | Harness command (function mode) |
{{MIN_COVERAGE}} |
config.thresholds.min_coverage_percent | Coverage gate (function mode) |
{{SAFETY_CRITICAL_FLOWS}} |
context.json | Flows requiring ASK-before-edit |
{{RECORD_FILE}} |
derived | .aid/pipeline/autoresearch/{{TASK_ID}}/results.tsv |
| pass-specific | config.autoresearch.<pass> | max_iterations, max_consecutive_reverts, internal rounds, and (AR-3) max_acceptance_rounds, pass_rate_target |
Per-pass caps come from config.autoresearch:
{ kpi_target, ar_design:{max_iterations,max_consecutive_reverts}, ar_function:{max_iterations,max_consecutive_reverts,internal_rounds}, ar_acceptance:{max_acceptance_rounds, pass_rate_target} }.
ar_designhas NOinternal_rounds. The bounded internal refine-rounds knob belongs toar_functionONLY (AR-2 allows a few internal apply/refine rounds before official scoring); AR-1 design does a single apply per iteration. Theinternal_roundskey was therefore removed from thear_designblock inconfig.default.json— it is a dead key underar_designand must not be reintroduced there.
Placeholder → config bindings
The per-pass prompt files may keep their {{...}} placeholder names; this table is the
authoritative binding of each to its config.autoresearch key:
| Placeholder | Config key |
|---|---|
{{AR1_MAX_ITERS}} |
config.autoresearch.ar_design.max_iterations |
{{AR1_MAX_REVERTS}} |
config.autoresearch.ar_design.max_consecutive_reverts |
{{AR2_MAX_ITERS}} |
config.autoresearch.ar_function.max_iterations |
{{AR2_MAX_REVERTS}} |
config.autoresearch.ar_function.max_consecutive_reverts |
{{AR2_INTERNAL_ROUNDS}} |
config.autoresearch.ar_function.internal_rounds (NEW key, default 2) |
The loop SKELETON (shared by all three modes)
The mode's per-pass file supplies the harness (how to produce the composite score), the rubric/dimensions, the composite formula, and the hard constraints. Everything below is identical across modes.
Setup phase (run ONCE per pass, then never again)
- Read the active per-pass prompt file for this mode (table above). It is authoritative for mutable surface, harness, rubric, composite, constraints, and stop conditions specific to the pass. The runner's stop conditions below are layered on top of those.
- Resolve the mutable surface. The artifact under optimization is the production source
changed by this task:
The union (filtered to source per the per-pass file) is your mutable surface. Everything else — the harness, tests (ingit -C "{{WORKSPACE}}" diff --name-only HEAD # tracked changes git -C "{{WORKSPACE}}" ls-files --others --exclude-standard # untracked (new) filesfunction/acceptance), criteria YAMLs,.aid/*.json, config — is read-only. (githere is read-only inspection; no commits, no resets, nocheckout/restoreof tracked files. Rollback is done via snapshot copies, below.) - Create the results directory and snapshots dir:
mkdir -p ".aid/pipeline/autoresearch/{{TASK_ID}}/snapshots" - Init
results.tsv({{RECORD_FILE}}) if absent or header-only, with this header line (tab-separated):
The per-pass dimension columns sit betweeniter composite delta <one column per rubric dimension> status snapshot utcdeltaandstatusand are named by the active per-pass file using the EXACT criterion ids (e.g. AR-1:structural_health loudness_of_failures naming_clarity docs_traceability; AR-2:quality pass_rate coverage). Each per-pass program file prints its OWN exact header. Single-letter abbreviations (S/L/N/D) are NEVER allowed as TSV column names — only in the human one-line chat output. Ifresults.tsvalready has data rows, do NOT re-baseline — resume from the next iter. - Score the baseline using the per-pass harness on the current on-disk source. Record the
composite as
best_score. Snapshot the baseline of every mutable file:TS="$(date -u +%Y%m%dT%H%M%SZ)" # Directory form — AUTHORITATIVE (matches each per-pass program file; per line 111 the # per-pass file wins): ONE baseline directory with the mutable files copied in preserving # their relative paths, so verify-run.sh's per-file revert check can diff against it. # Do NOT use a flat 000-baseline-<safe-F> file — that splits the baseline across files and # makes verify-run.sh skip the revert check (WARN, not FAIL). DST=".aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/000-baseline-$TS"; mkdir -p "$DST" # for each mutable file F: copy it into "$DST" preserving its relative path - Append row 0 to
results.tsvwithiter=0,composite=<best_score>,delta=0.00, the baseline dimension scores,status=baseline, the baseline snapshot tag, and UTC. - Record the rubric fingerprint (harness-tamper detection). At PASS START — once, here in
setup — compute a SHA-256 fingerprint over the EXACT bytes of the harness inputs that define
the score: the active mode's criteria YAML plus the reflection-agent's
{{WORKSPACE}}/.claude/agents/reflection-agent.md(concatenate the files in a fixed order, then hash). Record it instate.jsonunderstate.autoresearch.<mode>.rubric_sha256and note it in the stop summary. The whole point of the keep/revert loop is comparing scores across iterations against a FIXED harness — if the criteria YAML or the agent prompt change mid-pass, every score before and after the change is incomparable and the loop's keep decisions are meaningless. Thereforeverify-run.shrecomputes the same hash from the on-disk inputs and FAILS LOUDLY on any mismatch with the recordedrubric_sha256, rather than silently trusting a tampered run. (Spec-level: this runner is prose-driven — the runner computes/records the hash in prose andverify-run.showns the recompute-and-compare gate.)
Experimentation loop
Track an in-memory counter N (next iter from results.tsv) and consecutive_reverts.
For each iteration:
Read state. Re-read the on-disk mutable files. Read
results.tsv; setbest_score = max(composite where status='kept'), fallback baseline. Skim recentrevertedideas — do not repeat them.Pick ONE focused edit. A single, small, reviewable, single-purpose change permitted by the active per-pass file's constraints. One idea per iteration — never "while I'm here, also fix X" (multi-variable edits give uninterpretable scores).
Snapshot PRE-edit. Before touching anything, copy every mutable file you are about to change to a snapshot, so it is a clean rollback target:
SLUG="<short-kebab-of-the-idea>"; TS="$(date -u +%Y%m%dT%H%M%SZ)" cp "$F" ".aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/$(printf '%03d' $N)-pre-$SLUG-<safe-F>-$TS"Apply the edit. Edit only the mutable source. Some per-pass files allow a bounded set of internal refine rounds before official scoring — follow the file; default is one apply.
Score via the pass's harness. First, re-verify the harness fingerprint (mid-pass tamper guard): recompute the rubric SHA-256 (criteria YAML + reflection-agent
AGENT-PROMPT.md, same recipe as setup step 7) and compare it to the recordedstate.autoresearch.<mode>.rubric_sha256; on mismatch, HALT the pass loudly with statusrubric-tampered— every score across the change is incomparable — instead of waiting for the out-of-bandverify-run.sh. Then run the per-pass harness exactly as that file specifies (e.g.design: reflection-agent only;function:{{TEST_COMMAND}}+ coverage as Tier-1 gates × reflection quality;acceptance: validators + reflection). Compute the composite per the file's formula. This is the official iteration score.reflection-agent harness: spawn the subagent named
reflection-agentand load its prompt from{{WORKSPACE}}/.claude/agents/reflection-agent.md. NEVER hardcode a user-level path (e.g.~/.claude/skills/reflection/...) — it does not exist.Keep or revert (STRICT improvement).
- If
composite > best_score(strictly greater):status = kept; setbest_score = composite; leave the edited files on disk; resetconsecutive_reverts = 0. - Else:
status = reverted; restore each touched file from its step-3 snapshot (cp -p <snapshot> "$F"— preserve mtime so a restored file never reads newer thanstate.json);best_scoreunchanged;consecutive_reverts += 1. - If scoring crashed / harness errored (e.g. could not run tests):
status = crash; restore from snapshot (cp -p);consecutive_reverts += 1. - Restore robustness (every restore
cp -p): check the exit status of EVERY restorecp -p. If ANY restore fails, do NOT advance on a mixed/partial tree: set the distinct loud statusrestore-failed, append the row, and HALT the pass immediately (then jump to the stop sequence sostate.jsonis still written last). - Function mode only: a test going from green to red is an automatic revert regardless
of the quality sub-score (the per-pass composite already zeroes out via the Tier-1 hard
gate); log it as
status = reverted-redtest, restore from snapshot,consecutive_reverts += 1.
The
consecutive_revertscounter increments onreverted,reverted-redtest, andcrashalike (all three count toward the pass'smax_consecutive_reverts); only akeptiteration resets it to 0.- If
Append a
results.tsvrow:<N> <composite> <composite - best_score_before_this_iter> <dim scores...> <kept|reverted|crash> <snapshot> <UTC-iso>Print ONE chat line (see Output), increment
N, check stop conditions, loop.
results.tsv columns (authoritative)
This SKILL.md is authoritative for the COMMON columns, in this exact order:
iter composite delta <one column per rubric dimension> status snapshot utc
iter— iteration index; AR-1/AR-2 use an integer, AR-3 may use a string iter id (e.g.ar3-r<R>).composite— the pass's composite score.delta—composite - best_score_before_this_iter.<one column per rubric dimension>— the per-pass dimension columns, inserted betweendeltaandstatus. Each per-pass program file defines its OWN dimension columns and MUST print its exact header, using the EXACT criterion ids as column names:- AR-1:
structural_health·loudness_of_failures·naming_clarity·docs_traceability - AR-2:
quality·pass_rate·coverage - AR-3: per
program.ar-acceptance.md. Single-letter abbreviations (S/L/N/D) are allowed ONLY in the human one-line chat output — NEVER as TSV headers.
- AR-1:
status—kept|reverted|reverted-redtest|crash|restore-failed.snapshot— the pre-edit snapshot filename for this iter; AR-3 uses-.utc— ISO-8601 UTC.
Right-anchored positional parse rule (authoritative). The renderer/parser tolerates
per-pass dimension column sets by reading the common columns positionally from BOTH ends:
the FIRST column is always iter and the LAST three columns are always status snapshot utc, in that order, for EVERY pass — AR-1, AR-2, AND AR-3 alike. Everything between
iter and status is that pass's own columns: AR-1 and AR-2 lead with composite delta then
their rubric dimensions, while AR-3 has no composite/delta — its in-between columns are
exactly its two dimensions pass_rate why_alignment. A parser MUST key dimension columns off
the per-pass header row, NOT assume composite/delta at fixed positions 2–3. For this right-anchored rule to hold, a row
must NOT carry any trailing free-text field after utc: AR-3 in particular does NOT append an
inline one-line gap-summary field to its results.tsv row — every AR-3 pass ends, exactly like
AR-2, with status then snapshot (-) then utc as its final three columns. The human
one-line gap summary lives in chat/summary output ONLY, never in the TSV row. (program.ar-acceptance.md's
row spec MUST agree with this: trailing status, snapshot = -, utc.)
Stop conditions (bounded — the loop ALWAYS terminates)
Stop the loop when ANY fires. The first three caps come from config.autoresearch.<pass>;
all are enforced by the runner regardless of mode:
- Target met: the per-pass success condition holds (e.g. all KPI dimensions ≥
{{KPI_TARGET}}; forfunctionthe composite ≥ target with all Tier-1 gates green; foracceptancethe per-pass DONE condition — pass_rate ≥pass_rate_target, no blockers, why_alignment ≥ target). - Plateau:
consecutive_revertsreaches the pass'smax_consecutive_reverts. This counter increments onreverted,reverted-redtest, andcrashiterations; only akeptiteration resets it to 0. - Iteration cap:
Nreaches the pass'smax_iterations. - Cost brake: the pipeline-wide
cost_limits.max_per_run_usdis hit — stop immediately. - Human interrupt: any new prompt / interrupt.
- AR-3 outer loop:
acceptancemode additionally bounds its DEVELOP re-entry bymax_acceptance_rounds; on the cap with open blockers it ESCALATES to the user rather than looping the pipeline. (Details inprogram.ar-acceptance.md.)
On stop, the runner MUST, in this order: (1) restore the editable set to the best-kept
version (the highest-scoring kept snapshot, or the baseline if nothing was kept) — restore
each file with cp -p <snapshot> "$F" (preserve mtime so a restored file never reads newer than
state.json), and check the exit status of EVERY restore cp -p; if ANY restore fails, set
the distinct loud status restore-failed and HALT (do not advance on a mixed tree) — but
STILL proceed to write state.json last (step 3). Before the baseline fallback, test -r the
target snapshot first and surface an explicit error if it is absent or unreadable, rather
than cp-ing a missing path. (2) summarize: starting composite, final best_score, count of
kept vs reverted vs crash, the top kept changes by delta, and which stop condition fired;
(3) in pipeline mode, write the final scores to step_history and write
.aid/pipeline/state.json UNCONDITIONALLY as the FINAL action — after the last source mutation
of ANY kind (kept edit, revert/restore cp -p, or crash restore), so the Stop hook does not
re-block. Because a plateau stop often fires right after a run of reverts (whose restore cp -p
sets the source mtime to the snapshot's preserved mtime), the state.json write MUST be the last
disk write of the pass — never conditioned on the last write being a kept edit. Verify the
write symmetrically with the restore checks: check the exit status of the state.json write
(prefer write-to-temp then atomic rename) and re-read it to confirm it parses and its mtime ≥ the
newest source mtime. If the write fails, set the distinct loud status state-write-failed and
surface it — do NOT rely on the downstream Stop gate, whose "re-run /pipeline" message would
otherwise mask a genuinely failed write.
Safety brakes & hard constraints (non-negotiable — org + pipeline policy)
- NEVER auto-commit. NEVER run destructive git (no
commit,reset,checkout/restoreof tracked files,clean, force-push, branch deletion).git diff/ls-files/statusfor inspection only. Rollback is always via snapshotcp. The human commits after review; the runner only validates and leaves the existing "validated — your call" nudge. - Touch only the mutable surface resolved in setup. Never edit the harness, tests
(in
function/acceptance), criteria YAMLs,.aid/*.json,config.json,.mcp.json, or any pipeline/skill file. Modifying the harness makes the score meaningless. - Snapshots only under
.aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/, only via thecpcommands above. - No new dependencies, no framework migration, no plaintext secrets. If a fix seems to need a new dep or a secret, stop and surface it — do not add it.
- No scope reduction to game the score (don't delete behavior/criteria/tests to raise a number).
- One idea per iteration. Strict-improvement keep. Equal scores revert.
- Respect
{{CRITICALITY_PROFILE}}: ASK before edits touching{{SAFETY_CRITICAL_FLOWS}}. - Bounded by construction: per-pass
max_iterations+max_consecutive_reverts+ the cost brake (and AR-3'smax_acceptance_rounds) guarantee termination.
Output expectations (be quiet)
After each iteration, print exactly ONE line to chat:
iter <N> | <kept|reverted> | <composite> (Δ <delta>) | <one-line desc of the change>
Do not narrate reasoning or dump the reflection-agent's full breakdown. results.tsv is the
source of truth; the user can ask for detail. Emit the stop summary only when the loop ends.