layer-autoresearch
Contributor-only autonomous keep/discard experiment loop (after karpathy/autoresearch) that edits Flisk's L1-L4 Docker layer generators in a flisk checkout and keeps only changes that lower idle memor
SKILL.md.claude/skills/layer-autoresearch/SKILL.md
GitHubLayer autoresearch
Adapted from karpathy/autoresearch program.md. You are an autonomous researcher making the Flisk runtime layers cheaper without removing features.
Setup (with the user, once)
- Agree a run tag (e.g.
sep25). Creategit checkout -b autoresearch/<tag>frommain. The branch must be new. - Ask the user whether any tool auto-commits or pushes working-tree edits in this checkout. If one does, ask them to pause it for the run (or run from a separate
git worktree), because experiment commits and resets must not reachorigin/main. - Read the in-scope files in full:
- Editable:
runtime/engine/topologies/platform/docker/layers/*.star,runtime/engine/topologies/platform/docker/layers/prisma/*.star,runtime/engine/topologies/platform/docker/build/golden_image_dockerfile.star,runtime/engine/topologies/platform/docker/config/healthcheck.star. - Read-only ground truth:
scripts/autoresearch/layer-eval.sh,scripts/benchmark/container-scale.ts,tests/tilt-engine/test_runtime_container_footprint.py, and any spec for the runtime container footprint underopenspec/.
- Editable:
- The eval needs a Flisk project with at least
LAYER_EVAL_SAMPLE(default 20) services. It usesFLISK_BENCH_PROJECT, or../flisk-erp-systemnext to the checkout when that is unset. If neither exists, ask the user to clone the ERP scale-test project (about 100 services) once withflisk project erp --path ../flisk-erp-system(the eval generates and builds what it needs; do not start all of it withflisk up), or to pointFLISK_BENCH_PROJECTat another large project. - Create an untracked
autoresearch-results.tsvwith the headercommit mem_mib image_mib status description. - Confirm with the user, then start.
Experiment
Run scripts/autoresearch/layer-eval.sh > run.log 2>&1 (about 5-10 minutes). Read the results with grep -A20 '^---' run.log.
- Primary metric:
mem_per_service_mib, lower is better. This is idle memory per real service in the bench project, and it is what limits 100 services on 16 GB. - Tie-breaker: if memory is within 0.5 MiB of the best, a lower
unique_per_service_miborservice_image_mibwins, then simpler code. - Hard constraints.
status: okrequires all 20 services healthy, zero crashes, and passing feature tests. Anything else is a discard, or a crash if it didn't run. - Never edit the eval harness, benchmark, tests or spec. Never remove a user-facing feature: Prisma, Infisical entrypoint, hugo-as-feature, healthchecks, signal handling. Don't add dependencies.
Candidate ideas from earlier measurements (re-measure before relying on them):
- Duplicate
node_modulescopy in the L4 backend runtime (4.2 MB twice per service). - L1 installs
file,bashandopensslinto every runtime image (30 MB apk layer). Check what actually needs them first. - Bun runtime memory flags (e.g.
--smol) in the resolvedCMD. - Distroless or slimmer Bun base for L4.
- Per-service unique layer size.
Loop (never stop until the human interrupts)
- Note the current commit.
- Change the editable files with one idea.
git commit -m "<idea>".- Run the eval, redirected to
run.log(don't flood context). - If there's no
---block,tail -n 50 run.log. Fix trivial mistakes, otherwise logcrash. - Append a row to
autoresearch-results.tsv(untracked). - If the metric improved and
status: ok, keep the commit. Otherwisegit reset --hard HEAD~1. - An eval that runs longer than 20 minutes counts as a failure.
The first run is always the unmodified baseline.