Evolving with swarms

Each round, scripts/evolve.py runs K swarms in parallel. A swarm is N proposer calls, each reading one parent program and one focus x thinking-style prompt and replying with an idea of at most 45 words, then one writer call that turns the ideas and the parents into a complete program. The program is scored by your evaluator as soon as it exists and joins the archive the next round draws parents from: refine one of the best two, combine three of the best ten, explore two from anywhere, start fresh from the initial program. Every model call goes through headless claude -p, is priced at list rates from its modelUsage, and counts against a dollar cap.

The design is a text-only version of SwarmEvolve (Jiang and Karniadakis, arXiv 2611.08483), whose agents think inside an open model's activations, which an API model does not expose.

When it pays

Measured 2026-11-09 against stock OpenEvolve on Sonnet 5.5, three seeds per arm, three problems (references/evidence-2026-11.md):

	swarm (defaults)	Sonnet loop, best effort per problem
$ to 99% of reference, circle packing	$0.013–0.019	$0.32–0.44
$ to 99%, hexagon packing	$0.035–0.060	$0.53–0.58
$ to 99%, Heilbronn triangle	$0.09–0.50	$0.77–1.02
best at $0.25	above every paired Sonnet seed on all three
final score at $3	swarm stops at 60 min, about $1	the same finals, in 7–42 minutes

Use the swarm when the budget is small or the goal is "good, cheaply". When the last 0.1% matters, the money is there, and wall-clock time counts, a Sonnet loop at the problem's best effort level gets to the same place faster.

When NOT to use this skill

situation	use instead
hundreds of failing tests to drive to green	swarming-to-green
choosing a model or effort for one subagent or a cascade	agent-routing
many independent tasks with checkable outputs, not one program to improve	dispatching-subagents (headless runner)
quality only a person or a judge model can rate	no scorer, no search: write the scorer first, or use a judged loop
claude.ai chat, where there is no claude binary	orchestrating-chat-agents

Abandon a run mid-way when:

• status shows rejected_above_max climbing or a best score above anything the problem allows: the scorer has a hole and the search is now optimising the hole;
• the best score is flat for ten rounds and rounds.jsonl shows every refine swarm parenting the same program: the archive has converged, and more rounds buy nothing;
• last_quota.seven_day passes about 0.40: the gateway stops at 0.45 anyway.

Earned exceptions

rule	banned when	earned when
writer at effort medium	—	effort low on a problem already known to be easy (circle packing: low reached 0.999 cheaper)
N=8 proposers per swarm	the problem is new and N=0 has not been tried	an N=0 control on this problem lost to N=8 (on circle packing the writer alone nearly tied)
no sandbox prefix	the evaluator imports model-written code on a machine with secrets or network	a throwaway container with no credentials, and the user agreed

Setup

1. Make a problem directory. Copy assets/example/ and replace its four files:
   • problem.md: the task in plain words, including the reference value if known;
   • initial_program.py: a valid, weak starting program;
   • evaluator.py: evaluate(program_path) -> dict of numbers and strings, with the score under score_key (default combined_score) and an error key on failure;
   • config.toml: entry function name, the interface text the models see, budgets.
2. Score the initial program by hand and read the dict:

   python3 -c "import importlib.util as u; s=u.spec_from_file_location('e','evaluator.py'); e=u.module_from_spec(s); s.loader.exec_module(e); print(e.evaluate('initial_program.py'))"
   

   A score of 0 with no error, or a missing score_key, means the contract is wrong; fix it before spending anything.
3. Set score_max from a bound you can prove (a known optimum, an area or volume bound), slightly above it. Scores above it count as 0 and are flagged. Leave it unset only when no bound exists, and then re-check the best program independently.
4. Choose the sandbox. The evaluator imports and runs model-written code.
   • Claude Code on the web with the claude-workspace hub: the hub jail.

     [sandbox]
     prefix = ["sh", "/home/user/claude-workspace/scripts/jail.sh", "-C", "{run_dir}", "--"]
     chown_uid = 65534
     

     and put --run-dir under /tmp/jail/<name>/ (mkdir and chmod 755 it first). Launch through a script file: the hub's jail_route guard refuses a Bash command that names /tmp/jail/ next to python3.
   • Desktop with Docker: prefix = ["docker", "run", "--rm", "--network=none", "-v", "{run_dir}:{run_dir}", "python:3.12-slim"] (add numpy or what the evaluator needs to the image).
   • No sandbox only under the earned exception above.
5. Check the CLI: which claude (desktop: the app's bundled CLI may not be on PATH; set EVOLVE_CLAUDE_BIN). Then run the offline selftest:

   python3 scripts/test_evolve.py      # three "ok" lines, about 3 s, no model calls
   

Run

S=/mnt/skills/user/evolving-with-swarms/scripts
python3 $S/evolve.py run problem/config.toml --run-dir RUN --seed 1 --cap 1.0 --wall 60 > RUN.log 2>&1

Launch it with Bash run_in_background: true and end the turn. The completion line is done -> RUN/summary.json; relaunching the same command resumes from RUN/archive.json and keeps the spent dollars. Check in with python3 $S/evolve.py status RUN. In Claude Code on the web a background task is stopped after two hours: keep --wall under 110 minutes per launch, or relaunch on the completion notice.

knob ([swarm], [budget])	default	measured
K, N	4, 8	K=2,N=16 and K=8,N=4 did no better on circle packing; N=0 nearly tied
proposer_effort	low	the floor; Haiku still thinks on about a third of calls
writer_effort	medium	low helped on circle packing, hurt on hexagon and Heilbronn
writer_model	haiku	sonnet (one seed): 99% at $0.11 on circle packing, between swarm and loop
cap_usd, wall_minutes	1.0, 60	the swarm spent $0.9–1.4 an hour on 4 vCPUs

On a new problem, run the default and an N=0 control at the same cap before a long run.

Reading results

RUN/summary.json: programs, best_score, spent, usd_to (dollars when a program first scored 0.99, 0.999, 0.9999), calls_by_role, rejected_above_max, duplicates, no_code, last_quota. RUN/best_program.py is the top program. events.jsonl has one line per evaluation with the spend at that moment, which is the curve to plot; rounds.jsonl has each swarm's parents and ideas.

Common failure modes

• best_score far above the reference, or rejected_above_max > 0 → the scorer accepts something it should not. Read the offending program in RUN/r*/s*/prog.py, fix the evaluator or tighten score_max, and start a new run: the archive is poisoned.
• Every refine swarm in rounds.jsonl parents the same broken program → a scorer hole made it the leader; same remedy. Diagnosed 2026-11-09: a hexagon run spent its last 40 minutes refining a program the scorer had wrongly rated 3.9e9.
• no_code climbing → the writer's reply has no fenced block defining entry. Check that entry in the config matches the interface text and the initial program.
• duplicates close to the number of writer calls → the writer returns its parent unchanged; usually a writer that cannot see why the score is low. Add the useful metrics to show_metrics, or raise writer_effort.
• Evaluation errors with rc=125 or rc=2 → the jail could not isolate (125) or nobody cannot enter the run directory (2). Put the run dir under /tmp/jail/<name>/ and set chown_uid = 65534.
• Spend per program well above a cent → the gateway's ledger shows the cost by role; proposers were 30–39% of the spend in the 2026-11 ledgers, the writer the rest. Each claude -p call carries about 1,370 tokens of the CLI's own, so many tiny calls cost more than their prompts suggest. Fewer proposers (or N=0) and the writer's effort are the levers.
• pauses.jsonl appears and the run slows → the account's five-hour window passed 90% (non-Haiku calls) or 97% (Haiku). The gateway waits for the reset and leaves the wait out of the run clock; nothing to do unless it is urgent.
• A run stops with stopped_by starting quota: → seven-day utilisation reached 0.45. Stop launching runs this week, or raise week_stop in gateway.py deliberately.

Verification

python3 $S/test_evolve.py                         # offline: run, resume, status, score bound, sandbox prefix
python3 $S/evolve.py status RUN                   # programs > 1, spent <= cap + one round, best_score <= score_max

Re-score RUN/best_program.py with the evaluator yourself (in the sandbox) and compare it with best_score. Programs that optimise with randomness or a time budget re-score lower; two of three hexagon programs did on 2026-11-09. A bad success is a high score the evaluator should not have given: check the best program against an independent condition (a stricter geometric test, a known bound) before reporting a record.

Diagnosed failures

• 2026-11-09, hexagon packing. OpenEvolve's stock hexagon evaluator scored an outer side of 1e-9 at 3.9e9: an absolute tolerance on an unnormalised cross product. Signal: a score nine orders above the reference. Fixed by an area bound; score_max exists for this.
• 2026-11-09, cost accounting. claude -p wrote prompt cache at the 1-hour TTL (2x input) until FORCE_PROMPT_CACHING_5M=1; the gateway sets it. Signal: cacheCreationEphemeral1hInputTokens non-zero in the ledger.
• 2026-11-09, effort. MAX_THINKING_TOKENS=0 and alwaysThinkingEnabled: false did not stop Haiku 5.5 from thinking, and an output cap below its need failed the call while billing it. Effort low is the floor; do not set max_tokens tight.
• 2026-11-09, usage window. Ten parallel runs used most of a five-hour window in about 70 minutes and the old gateway aborted runs at 90%. It now pauses and resumes instead.