Model validation

Read docs/reference/qualification-runner.md for the current supported CLI and docs/reference/qualification-transport.md for provider identity and receipt rules. Inspect the shipped routing alias and effective request settings; do not invent a candidate catalog or assume advertised parameter support proves provider behavior.

Recover before recreating

Reconcile the private checkpoint with actual PR merges and raw verification evidence. A stale handoff's blocked commit may already be merged. Verify exact source/input bindings before reusing prior acceptance receipts; run the specific missing checks rather than automatically repeating the entire synthetic campaign. Preserve earlier failures and source generations.

Keep one private checkpoint with the chosen source commit, actual approved corpus/policy identities, fixed sample, execution plan, actual consent and remaining decisions. Keep credentials, references and raw answers off public surfaces. Do not add this external client's transcript or identifiers to Genesis session tools.

Admit the complete study offline

Use a dedicated serial process, disk-backed disposable SQLite and the shared lock/resource limits for offline acceptance. Never boot the serving runtime from a worktree. Pin source hashes and use the actual production rendering, target selection and both storage entry points. Check retained source originals and the independently restored backup required by the approval metadata.

The complete current corpus must pass admission even when an incumbent uses a sampled subset. Preserve human and frontier label provenance, uncertain/excluded originals and actual human decisions. New human guidance needs a reasoned applicability check across all cases, including previously accepted cases. Changed inputs or policy invalidate the affected approvals; do not restart grading for unchanged approved bytes.

check performs offline admission. report uses retained answers without credentials or provider requests. run requires an operating plan and separate spending approval. archive-sources retains source bytes; it does not create human approval or prove inclusion in an external backup. Read the current command parser before constructing arguments.

Freeze the exact paid decision

Count every planned case, model and repetition. If sampling, freeze case IDs and class strata before observing model results; no automatic resampling or expansion after a close result. An incumbent sample is a baseline, not full requalification.

Compute conditional input/output costs from the actual rendered messages, current public price evidence and explicit framing/reasoning assumptions. Keep operating allocations distinct from provider-verified charges and a hard invoice ceiling. A shared key measures this experiment's receipts and pending allocations, not unrelated account spending or exclusive account headroom.

Independently recompute each request allocation from the tokenizer artifacts, raw prices, actual wire messages and output caps, stated framing assumptions, margin and currency rounding. Summing the builder's saved allocations is not an independent arithmetic check. Include unresolved liabilities from previous campaigns in the stated total target without treating them as zero or silently clearing them.

Freeze the gateway model, selected backend_model and generation receipt_model separately when the provider uses different namespaces. Use independent endpoint/receipt evidence for exact IDs; state any unobserved receipt expectation and fail closed if it disagrees. Do not strip date suffixes or choose aliases from a candidate's answer. Preserve raw identities when deriving a canonical accounting model.

When no authorized fingerprint exists, obtain narrow local key-identification permission before selecting/reading/hashing the existing key. This permission includes no HTTP or spending. Then freeze the complete plan and obtain actual paid-use approval bound to its canonical hash and exact request-count scope. Design approval, corpus approval and synthetic fixtures cannot supply that consent. Changes to the bound plan require the corresponding new decision.

Acquire once and preserve uncertainty

Use the existing operating controller, actual delegate and exact serialized-request checks. Keep both campaigns in an isolated process with one inference in flight. Preserve dispatches, observations, answers, generation IDs, receipts and incidents before any subsequent request. Stop the selected population on unresolved or contradictory evidence or an above-allocation charge.

Do not automatically retry, resend, top up, expand the sample or reconstruct a lost answer. GET-only receipt recovery cannot recreate an answer or permit a second dispatch of that coordinate. Acknowledgement records verified resolution without deleting history. An incomplete campaign cannot qualify a model. Never change account limits or production routing as a side effect of qualification.

If a code repair changes the frozen source or identity pins, preserve the stopped campaign under its old manifest. Re-admit the actual merged source and prepare a new hash-bound plan. Any replacement of a historical dispatched coordinate must be explicitly included in the owner's new paid decision; repair or merge approval does not authorize it. Keep the original fixed sample unless the owner makes a separate study-design decision.

Use a durable exclusive launch marker before acquisition, and define how readers treat partial preparation, a killed child and missing terminal receipts. An existing marker, partial output or campaign directory prevents automatic relaunch. Read-only reporting can resume; acquisition cannot be used to repair an incomplete report.

Report the measured result and close the evidence

Separate complete candidate qualification from a sampled incumbent baseline. Report contract/class denominators, each repetition, malformed grades, completeness, actual expenses, unresolved liability, latency and requested/observed identity and controls. For paired comparisons use the same declared sample; repeated answers do not create independent cases. Keep exploratory statistics and source-family correlation explicit. Allow an inconclusive outcome.

Synthetic embeddings, extraction stubs and mock HTTP establish integration, not model quality or retrieval quality. A constructed diagnostic novelty corpus does not certify safe production suppression or rare-error absence, even with no observed suppression errors. Promotion needs separate role-specific evidence and an owner decision, qualified fallback and rollback; deployment is a further authorized operator action.

Retain the completed plan, consent, raw journals, report and source identities. Verify backup inclusion and an isolated restore for irreplaceable evidence. Reconcile verification obligations through the authorized validator; ordinary external-client access does not grant verification-write or deployment authority.