Recording runs

Every real run gets logged — pilot, sweep, failed or not. Skip only throwaway smoke tests (n < 10, just checking code runs); delete their output. A run on real data at a meaningful n that produces a number anyone might compare against is a real run.

Steps

1. Save the record with RunRecord:

   from latentreasoning.runlog.manifest import RunRecord, ModelInfo, DatasetInfo, new_run_id
   from latentreasoning.utils import set_seed
   
   seed = set_seed(42)                       # before any training/generation
   ...
   record = RunRecord(
       run_id=new_run_id("filler_tokens", "budget-sweep-32"),
       mechanism="filler_tokens",            # latentreasoning.runlog.manifest.MECHANISMS, or "baseline_*"
       stage="pilot",                        # "smoke_test" | "pilot" | "full_run"
       model=ModelInfo(backbone="gpt2-edited", checkpoint="results/<run_id>/ckpt", n_params=124_439_808),
       dataset=DatasetInfo(name="gsm8k-aug", split="test", n_examples=200, seed=0),
       metrics={"final_answer_accuracy": 0.31, "unparseable_rate": 0.02, "compute_steps": 32,
                "extra": {"cot_tokens": 41}},  # mechanism-specific stuff goes under "extra"
       hyperparams={"lr": 5e-5, "epochs": 3, "batch_size": 16, "filler_token": "."},
       seed=seed,
       hardware="RunPod A40 x1",
       notes="one line on what this run asks",
   )
   record.save(predictions=eval_result.records)   # -> results/<run_id>/{manifest.json, predictions.jsonl, notes.md}
   

   Always pass predictions — per-example outputs are the error-analysis and decoder-training data later. Also drop trainer.state.log_history into results/<run_id>/train_log.json if you trained.

2. Fill in results/<run_id>/notes.md — save() created a template with the numbers pre-filled. Goal, command, interpretation, gotchas, caveats, next. A number without the story is hard to trust later.

3. Regenerate the index: uv run python scripts/rebuild_index.py rewrites results/README.md and EXPERIMENTS.md. Never hand-edit those two.

4. Commit results/<run_id>/ + the two generated files. Checkpoints and other large artifacts stay on the pod volume (.gitignore drops everything in results/<run_id>/ except manifest / notes / predictions / train_log); reference them by path in model.checkpoint.

Never rewrite a past run's numbers or notes. If it turns out flawed, append a correction to its notes.md (or log a follow-up run) and rebuild.

Merge conflicts

Run directories never conflict. results/README.md and EXPERIMENTS.md will — take either side, rerun rebuild_index.py, commit. tests/test_index.py fails if they're stale.

Comparing runs — the compute-budget rule

metrics["compute_steps"] is the budget knob: filler tokens emitted / loop iterations r / continuous-thought tokens. explicit_cot is unbudgeted; it reports metrics["extra"]["cot_tokens"] instead.

Only compare final_answer_accuracy when compute_steps, dataset.split, dataset.n_examples, dataset.seed, and seed match. 32 filler tokens vs r=32 loops is not automatically matched — say in notes.md what you matched on (extra forward passes, FLOPs, ...) before quoting one against the other.

from latentreasoning.runlog.manifest import load_all_runs
runs = [r for r in load_all_runs() if r["dataset"]["split"] == "test"
        and r["dataset"]["n_examples"] == 200 and r["metrics"].get("compute_steps") == 32]

Shell: jq '[.mechanism, .metrics.final_answer_accuracy, .metrics.compute_steps]' results/*/manifest.json

Runs on unmerged branches aren't in main's index — check open branches before concluding something wasn't tried.

Canonical metric keys

final_answer_accuracy, unparseable_rate (don't let a high one hide inside a mediocre accuracy), compute_steps, train_loss, eval_loss, sec_per_example. Anything else under metrics["extra"].

Reserved for the October milestones (PROPOSAL.md metrics) -- use these spellings when you get there, and define exactly how you computed it in notes.md: decoding_accuracy, early_termination_necessity, intervention_accuracy, transplant_success. Note early_termination_necessity (truncate the scratchpad at inference on a full-budget model) is a different measurement from a compute_steps sweep (train + eval at each budget) -- don't quote one as the other.