ate
ATE (Automated Testing Environment): drive long-running (2-3hr+) unattended variation-test suites against the Aider, OpenRouter, and GLM 5.2 pipeline using a local Gemma model in LM Studio as the work
SKILL.mdskills/4-occasional/ate/SKILL.md
GitHubATE — Automated Testing Environment (generic bounded-variation matrix runner)
The skill interface lives here; its implementation is bundled under
utils/ate/.
ATE walks a declared grid of command variations unattended for hours, logs every result as
structured JSON, and a frontier model (you, Claude) checks in every ~5 minutes to catch
drift/looping. The full cycle — run, capture, document, file — is chained end to end: when the
run ends (time limit, abort, or safety cap), run_variations.py automatically hands off to
compile_issue.py, which opens one GitHub issue titled
ATE - [test-name] yyyy-mm-dd containing every finding from that run as a single
checklist, ranked by severity (critical first). No manual second step required
as long as --gh-repo was passed.
For Radar-selected defect discovery, use defect-campaign for suitability, bounded approval and replay/disposition requirements. In that mode, synthesis/healing and automatic filing stay off; this skill's older filing flow does not authorize new CI cases or remediation.
Architecture (linear)
- Gemma (LM Studio, local) — worker. Cycles through the variation grid
(repeating from the top once it reaches the end), runs
aider --model openrouter/z-ai/glm-5.2 ...as a subprocess, classifies the result. - error_log.jsonl — the single source of truth. Append-only. Every variation writes one line here regardless of pass/fail.
- control.json — the only channel Claude writes to. Gemma polls it before each
iteration.
{"action": "continue"}/{"action": "abort", "reason": "..."}.run_variations.pyresets this tocontinueat the start of every run, so a stale abort from a previous run in the same directory can't kill a fresh one. - Claude (you) — supervisor. Every ~5 min, run
checkin.py, read the summary, decide continue/abort, writecontrol.jsonif aborting. - compile_issue.py — chained automatically:
run_variations.pycalls it the moment the run stops (time limit, abort, or safety cap), passing--gh-repothrough. It groupserror_log.jsonlby severity/signature into one unified, severity-ranked checklist and opens a single GitHub issue titledATE - [test-name] yyyy-mm-ddviagh issue create. Can still be run standalone for a manual rollup (e.g. after a crash that killedrun_variations.pybefore it could chain).
Quick start (do this first)
# 1. Install
bash utils/ate/install.sh # copies the interface + implementation to ~/.claude/skills/
# 2. In LM Studio: load a Gemma 4 model (31B Dense recommended), start the
# Local Server (Developer tab -> Start Server). Default: http://localhost:1234/v1
# Note the exact model identifier shown in LM Studio's server log/model list.
# 3. Set OpenRouter key for the Aider side of the pipeline being tested
export OPENROUTER_API_KEY="sk-or-v1-..."
# 4. Point run_variations.py at a scratch git repo (never your real repo).
# error_log.jsonl and control.json land in this directory by default —
# run checkin.py from here too, or pass --log/--control explicitly.
cd ~/scratch/aider-test-repo && git init -q
# 5. Kick off the run (defaults to 3 hours, edit variations.yaml first).
# --gh-repo is what turns on the automatic issue filing at the end — omit it
# to just get error_log.jsonl with no GitHub side effect. Requires
# `gh auth status` to already be logged in.
python3 ~/.claude/skills/ate/scripts/run_variations.py \
--repo . \
--variations ~/.claude/skills/ate/variations.example.yaml \
--lmstudio-model "gemma-4-31b-instruct" \
--gh-repo OWNER/REPO \
--test-name "aider-flag-fuzz" \
--minutes 180 &
Then, on your (Claude's) side, every ~5 minutes:
python3 ~/.claude/skills/ate/scripts/checkin.py --tail 20
Read the printed summary. If it looks healthy, do nothing (Gemma keeps polling
control.json, sees no abort, continues). If you see drift (see "What counts as
drift" below), abort the run — checkin.py --abort "reason" does this for you in
one call. There is no "redirect" action; the only control signals are continue
and abort.
When the run ends (time limit, abort, or the iteration safety cap of 20 full
passes over the grid — see "Notes / gotchas"), run_variations.py automatically
chains into compile_issue.py and opens one GitHub issue titled
ATE - [aider-flag-fuzz] 2026-07-09 (using today's date and the --test-name
you passed, or the variations.yaml filename's stem if you didn't pass one) with
every finding from the run as a single severity-ranked checklist. Nothing further
to run by hand. If you need the standalone/manual form (e.g. run_variations.py
itself crashed before it could chain, or --gh-repo was omitted):
python3 ~/.claude/skills/ate/scripts/compile_issue.py \
--log error_log.jsonl --repo OWNER/REPO --test-name "aider-flag-fuzz"
Exit codes (#142)
Both scripts in the filing chain exit with a contract a wrapper, CI job, or supervising agent
can branch on. Before #142 every terminal state exited 0 — including a failed gh issue create
— so a multi-hour run's final step could fail invisibly.
| Code | Meaning |
|---|---|
0 |
Issue filed, or --dry-run body rendered. |
3 |
No records in the log — nothing to file. Distinct from filed so callers can tell "done its job" from "did nothing". |
1 |
gh issue create failed. issue_body.md is preserved in the working directory for manual filing. |
2 |
run_variations.py only: repo guard refused the run (--repo is the harness itself, or carries a remote without --allow-destructive-reset). |
run_variations.py propagates compile_issue.py's code when --gh-repo is set: the run's exit
code IS the filing outcome. Supervisors: treat 0 and 3 as healthy ends, 1/2 as needing
attention, and abort-drift separately via checkin.py. The gh issue create call is capped at
120s (ATE_GH_TIMEOUT_S overrides); a hang, a missing gh, or a launch failure all land in
exit 1 with issue_body.md preserved — no traceback, no indefinite hold on the run's completion.
Targets beyond Aider (#141 Phase 5)
The engine is target-agnostic: any grid with a command_template (argv-list form — values are
substituted per-token, never through a shell string) runs against any CLI. Aider is one preset,
not the subject.
- Classifier oracle decoupling (
expects_edits) — the stock Aider grid expects the tree to change, so exit-0-with-no-edit classifiesfail/no_edit. A grid that only probes (usage surfaces, read-only diagnostics) declaresexpects_edits: false, and exit-0-no-edit is a PASS. The #146 Gemma soak recorded 17 false HIGHno_editverdicts before this key existed — do not run a non-edit grid without it. - Rollup labels are neutral by default —
bug, nothing else. The Aider preset opts back in viaissue_labels: [bug, aider-pipeline]in its grid; any grid or--issue-labeloverrides. A turn-shim soak must never file Aider-labelled issues. - Declared non-Aider grids shipped here:
variations.tool-density.yaml/variations.tool-calling.yaml(script_runner benchmarks) andvariations.turn-shims.yaml(turn-shim CLI-contract usage soak — read its header for the safe execution profile before wiring it to a runner; it is diagnostic,expects_edits: false, and must run against stubs or a disposable clone if extended past the usage surface).
What counts as drift (abort triggers)
- 3+ consecutive iterations with the same
likely_causesignature — Gemma is stuck re-testing the same bug instead of moving to the next variation - Gemma marks something
passbut the raw Aider stdout/stderr in the same record contains a traceback or non-zero exit code — classification is unreliable, stop and inspect - No new lines appended to
error_log.jsonlfor 2+ check-in cycles — the worker has hung (check the LM Studio server and theaidersubprocess).checkin.pyhas no memory of previous invocations, so use its printedlog_last_modified_seconds_agofield and compare it against what you saw last check-in rather than relying on the tool to flag this automatically. - Wall-clock time on a single variation exceeds
--per-variation-timeout(default 180s) — treat as a hang, not a slow test;run_variations.pykills it and logs it asseverity: high, category: timeoutautomatically
variations.yaml format
Grid of Aider flags to combine (see variations.example.yaml). Each combination
becomes one subprocess call. Keep the task message short and deterministic (a
fixed small task like "add a docstring to foo()") so failures are attributable to
the pipeline (Aider/OpenRouter/GLM plumbing), not to task ambiguity.
model: openrouter/z-ai/glm-5.2
edit_formats: [diff, whole, udiff]
map_tokens: [0, 1024, 4096]
auto_commits: [true, false]
message: "Add a one-line docstring to the function `foo` in sample.py"
per_variation_timeout_seconds: 180
Severity rubric (used by compile_issue.py to rank the unified checklist)
| Severity | Definition |
|---|---|
| critical | Aider process crashes, non-zero exit with traceback, or corrupts the git working tree |
| high | Wrong/no edit applied, OpenRouter auth or routing failure, timeout/hang |
| medium | Edit applied but malformed (bad diff format, partial file write) |
| low | Cosmetic — extra output noise, formatting drift, slow-but-succeeded |
Files in this skill
scripts/run_variations.py— the Gemma-driven worker loop (long-running); chains intocompile_issue.pyautomatically when it stops, if--gh-repowas passedscripts/checkin.py— Claude-side supervisor: summarize + optionally abortscripts/compile_issue.py— rolls uperror_log.jsonlinto one GH issue titledATE - [test-name] yyyy-mm-dd; runs standalone too, for a manual rollupvariations.example.yaml— starter grid, copy and edit per pipeline under testutils/ate/install.sh— copies the interface and implementation to~/.claude/skills/
Notes / gotchas
- LM Studio's OpenAI-compatible endpoint ignores auth — no API key needed locally,
but you must pass the exact model name LM Studio reports (
/v1/modelswill list it). - Aider's OpenRouter model string needs the
openrouter/prefix on top of the OpenRouter slug itself:openrouter/z-ai/glm-5.2— the bare slug alone will fail. - Gemma 4's training cutoff is Jan 2025 — if Aider's CLI flags have changed since,
Gemma may generate stale flag names.
run_variations.pyonly ever calls flags fromvariations.yaml(never lets Gemma invent flags), which sidesteps this. - Run everything against a disposable scratch repo.
run_variations.pydoes a destructivegit reset --hard+git clean -fdxbefore every variation (so each variation starts from the same pristine state and results are comparable across the grid). This permanently deletes uncommitted and ignored files in--repo. Two guards refuse the obvious footguns before anything runs (GH-195): it hard-refuses--repopointed at the harness clone itself, and refuses a--repothat has a git remote (i.e. looks like a real checkout) unless you pass--allow-destructive-reset. A freshgit initscratch repo has no remote, so the Quick start above needs no extra flag — and agit initwith no commits yet no longer crashes (the script auto-creates a base commit to rewind to). - The grid is cycled, not walked once —
run_variations.pyloops back to the start after the last combination and only stops on the time budget, an abort, or an iteration safety cap oflen(combos) * 20(a backstop against a runaway--minutesvalue).