durable-hpc-gate-ecoda
Applies durable-hpc-gate to ECODA on Yggdrasil, adding repository invariants, authoritative-plan checkpoints, canonical scratch/NAS contracts, clean benchmark serialization, and Luna Max task/reviewer
SKILL.md.agents/skills/durable-hpc-gate-ecoda/SKILL.md
GitHubDurable HPC Gate — ECODA/Yggdrasil Profile
Overview
This is the ECODA policy layer for the global durable-hpc-gate skill. It
keeps long-running Slurm wrappers on Yggdrasil durable across SSH/session loss
without copying the global orchestration implementation. The profile is
repository-aware: it checks the ECODA checkout, canonical configuration and
artifact roots, scientific invariants, benchmark wave serialization, and the
terminal accounting/audit contract.
Canonical host policy
yggdrasil is the current default and canonical host for full-cohort ECODA
Pipeline 1–5 durable compute, data, results, backup, and normal gates until
the user explicitly directs a return to Bamboo. All authoring and
pipeline-file changes happen on the local workstation: commit and push them,
then pull the exact committed revision on Yggdrasil. Never edit the Yggdrasil
checkout directly.
The active Yggdrasil repository and scratch paths are ~/ECODA_paper and
~/scratch/ECODA_paper, respectively. Bamboo is an explicit user-directed
fallback/source host only; do not infer a host change from maintenance dates
or use Bamboo paths by default. The current checked-in
references/profile.json and effective workflow configuration MUST be
Ygg-compatible before any gate runs, including remote_host, repository and
scratch roots, scheduler/runtime settings, and result handling. A profile
that still encodes Bamboo as its active host is not valid for a Yggdrasil
gate.
Yggdrasil has no implicit Bamboo NAS path. NAS_TARGET_DIR (or an explicit
ECODA_NAS_TARGET_DIR override) MUST point to a configured Yggdrasil result
location, such as the approved local scratch result mirror, or a verified
user-scoped NAS mount MUST be established before launch. Never silently fall
back to Bamboo NAS storage. Any Bamboo↔Yggdrasil transfer is exceptional and
requires an explicitly configured SSH key or agent-forwarding setup. Every
transfer/backup command MUST state its host, source/destination path, and
scope explicitly; never store passwords, and use SSH keys/agent
authentication only.
Repository-defined exceptions
For this checkout, AGENTS.md is authoritative for explicit user-requested
exceptions to the durable-gate requirement. Keep exception criteria in
AGENTS.md rather than duplicating them here; follow its exact scope and
output restrictions, and ask if they are unclear.
The profile contains no credentials, email addresses, scheduler secrets, or
alternate runtime. Use the global CLI and the checked-in profile at
references/profile.json; do not add a second runner or wrapper here.
Dependencies
durable-hpc-gate(required runtime dependency): supplies the stdlib-onlyuv runCLI, durable manifest, remote runner, event-driven waiter, monotonic lifecycle, and terminal inspection.workflow-skill-creator(authoring only): used to structure this skill; it is not needed to run an ECODA gate.
Quick Start
Run these commands from the local ECODA repository. All authoring and
pipeline-file changes must be made locally, then committed and pushed; before
any gate runs, Yggdrasil must pull that exact committed revision. Never edit
the Yggdrasil checkout directly. Substitute a new gate ID, wrapper, and
artifact paths for the gate being run. Resolve Yggdrasil's home directory
remotely; local $HOME is not a valid prefix for remote artifact paths. The
remote checkout is ${YGGDRASIL_HOME}/ECODA_paper; its scratch data root is
${YGGDRASIL_HOME}/scratch/ECODA_paper (not a git clone). The checked-in
profile must be Ygg-compatible, and NAS_TARGET_DIR must be explicitly
configured for Yggdrasil or backed by a verified user-scoped mount; never
silently use a Bamboo NAS path. The exact wrapper must invoke the immutable
snapshot's own executor with its source manifest, frozen tree/aux root,
versioned runtime image/manifest, run ID, separate logs root, and explicit
selected-row arguments.
GLOBAL="$HOME/.agents/skills/durable-hpc-gate/scripts/durable_hpc_gate.py"
PROFILE="$PWD/.agents/skills/durable-hpc-gate-ecoda/references/profile.json"
GATE_ID="ecoda_<unique-id>"
MANIFEST="$PWD/.gate/${GATE_ID}.json"
YGGDRASIL_HOME="$(ssh yggdrasil 'printf %s "$HOME"')"
REMOTE_WORKDIR="${YGGDRASIL_HOME}/ECODA_paper"
WRAPPER='/bin/bash "<resolved-snapshot>/tree/src/utils/bash/ecoda_source_snapshot.sh" exec --source-root "<resolved-snapshot>/tree" --source-manifest "<resolved-snapshot>/identity/source.manifest" --host-env-prefix "<pinned-env-prefix>" --runtime-image "<versioned-runtime>.sif" --runtime-manifest "<versioned-runtime>.sif.manifest" --run-id "<run-id>" --scratch-root "<scratch-root>" --logs-root "<logs-root>" --script "src/<stage>/<submitter>.sh" -- <explicit-selection-arguments>'
REMOTE_ROOT="${YGGDRASIL_HOME}/scratch/ECODA_paper/gates/${GATE_ID}"
REMOTE_RUNNER="${REMOTE_ROOT}.runner.sh"
uv run "$GLOBAL" prepare \
--profile "$PROFILE" --manifest "$MANIFEST" \
--output "$PWD/.gate/${GATE_ID}.prepare.json" \
--project ECODA --gate-id "$GATE_ID" --remote-host yggdrasil \
--remote-workdir "$REMOTE_WORKDIR" --exact-command "$WRAPPER" \
--serialization-group ecoda-benchmark \
--tmux-session "$GATE_ID" --completion-channel "file:${REMOTE_ROOT}.done" \
--remote-manifest "${REMOTE_ROOT}.manifest.json" \
--remote-runner "$REMOTE_RUNNER" \
--remote-log "${REMOTE_ROOT}.log" --remote-status "${REMOTE_ROOT}.status.json"
uv run "$GLOBAL" reconcile --manifest "$MANIFEST" --profile "$PROFILE" \
--output "$PWD/.gate/${GATE_ID}.reconcile.json"
uv run "$GLOBAL" launch --manifest "$MANIFEST" --profile "$PROFILE" \
--output "$PWD/.gate/${GATE_ID}.launch.json"
serialization_group is the required explicit cross-manifest mutex identity.
Launch takes a deterministic lock keyed by project, profile, profile digest,
and group; an active RUNNING sibling with that identity records a
resource_lock_conflict discrepancy, moves this gate to PRELAUNCH_STOP, and
is not launched. Distinct groups may be prepared and launched in parallel,
including gates that share a reviewed predecessor. Reviewed dependency lineage
is independent of serialization_group.
After launch returns, the harness owns exactly one durable waiter. Start it
as an asynchronous, unbounded call (async=true, timeout=0 in the harness;
do not keep a task agent alive and do not send periodic polling requests):
uv run "$GLOBAL" wait --manifest "$MANIFEST" \
--output "$PWD/.gate/${GATE_ID}.wait.json"
When the completion event wakes the workflow, a fresh bounded Luna Max task runs one terminal inspection and records the evidence. Before that invocation, the task reads the durable wrapper log and extracts every scheduler array or watchdog ID the wrapper emitted after launch; it passes those values explicitly on the first inspect and never edits the manifest by hand. A passing audit is not release-eligible until the Luna Max reviewer explicitly approves it through the second inspect invocation:
uv run "$GLOBAL" inspect --manifest "$MANIFEST" --profile "$PROFILE" \
--scheduler-array-id "$ARRAY_ID" \
--scheduler-watchdog-id "$WATCHDOG_ID" \
--output "$PWD/.gate/${GATE_ID}.inspect.json"
uv run "$GLOBAL" inspect --manifest "$MANIFEST" --profile "$PROFILE" \
--approve-reviewer "Luna Max" \
--output "$PWD/.gate/${GATE_ID}.review.json"
Pass each emitted ID with a repeatable option; omit an option when that class
of ID was not emitted. The first inspect token-validates and atomically
deduplicates supplied IDs before setting audit_state: IN_PROGRESS. Supplying
IDs after audit begins or completes is rejected.
Use status for a non-mutating local/remote view. It is not a scheduler
poll and cannot replace inspect:
uv run "$GLOBAL" status --manifest "$MANIFEST" \
--output "$PWD/.gate/${GATE_ID}.status.json"
Every subcommand requires --output; output files are the structured record,
while stdout stays concise. If a command cannot prove the launch identity or
terminal state, stop and reconcile rather than retrying the wrapper.
Utility Scripts (if CLI-based)
The local profile has no utility scripts. All six approved subcommands belong to the global CLI:
reconcile: read-only adoption/recovery of the manifest against remote tmux/process/status and profile invariants; never launches or polls Slurm.prepare: validates the explicit gate specification and writes aPREPAREDmanifest atomically.launch: reconciles first, repairs only monitoring infrastructure when safe, and launches the exact command once in persistent tmux.wait: blocks on the completion event, then treats durable status as the authority if event transport reconnects.inspect: is legal only after terminal durable status; performs the single accounting query and profile artifact/checksum audits.status: reads durable state without mutation or scheduler polling.
Do not add aliases, implicit defaults, a second waiter, or a local copy of the Python implementation.
Workflow
1. Check the authoritative plan before preparing
- Use the existing authoritative
.kiloexecution-plan checkpoint. Record the gate ID, exact immutable wrapper, opaque serialization group, manifest, and profile. - On successful launch set that checkpoint to
RUNNING; after terminal inspection set it toCOMPLETEDorFAILEDwith links to status and audit evidence. Preserve the discrepancy history. - Never create a competing plan, edit the current plan during skill
installation, or mark a gate complete before
inspectand review.
2. Enforce Yggdrasil and repository invariants
- Remote host is
yggdrasil; the durable gate'sremote_workdirremains the repository clone$HOME/ECODA_paper. The clone is not$HOME/scratch/ECODA_paper; all source changes reach this checkout only through the exact local commit/push and Yggdrasil pull workflow. The exact wrapper executes the/bin/bashexecutor from the commit-keyed immutable source snapshot. - Create one verified source snapshot per full source commit before gate
prepare. It contains the complete committed tree,COMPLETE,identity/source.manifest,identity/source.tar, and the trackedtree/auxdirectory as the sole frozen auxiliary root. New wrappers exportECODA_SOURCE_ROOT,ECODA_SOURCE_MANIFEST,ECODA_SOURCE_SNAPSHOT_REQUIRED=1, andECODA_AUX_ROOT; they never source the mutable canonical checkout. - Each new run records exact run-bound copies of the source manifest and
runtime.identityunder its ownmanifests/directory, together with the selected-row/dependency metadata. Runtime image and runtime-manifest paths, digests, sizes, dependency fields, and permissions are bound to that run; writable logs live in the separately named run log root, not in read-only source ortree/aux. - Source
src/slurm_config.shfrom the immutable snapshot for every HPC wrapper. It remains the authority forPROJECT_ROOT,HPC_SCRATCH_DIR,NAS_TARGET_DIR,PYTHON_BIN,PIXI_RSCRIPT, scheduler resources, and retry ceilings. Snapshot-backed format-2 runs use the relocated immutable source/runtime layout; format-1 images and runs retain their legacy validation path. - On Yggdrasil, scratch data belongs under
$HOME/scratch/ECODA_paper; NAS artifacts belong under the explicitly configuredNAS_TARGET_DIR. That variable MUST resolve to an approved Yggdrasil result location, such as the explicitECODA_NAS_TARGET_DIRlocal scratch result mirror, or to a verified user-scoped NAS mount. There is no implicit Bamboo NAS fallback. Do not hard-code a second path or treat scratch as a git checkout. The frozentree/auxis the only source of auxiliary inputs, and Stage 4 validates (rather than rewrites) its pre-existing scGate database. - Workers use the immutable config-provided interpreter commands; never invoke
bare
python,Rscript, or an ordinary lock-mutatingpixi runin a job. Login nodes are for editing, staging, synchronization, and submission; heavy preprocessing/benchmarking runs in Slurm. - New runs do not couple identity to mutable canonical-checkout
git HEADor livepixi.toml/pixi.lockfingerprints. Source/config/dependency identity comes from the verified source manifest and the run-bound runtime identity; preserve the checked-inpixi.toml, resolvedpixi.lock, anddatasets.jsonunless explicitly authorized. - Before terminal inspect, the completion task invokes only the exact
run-scoped validator
ecoda_run_audit.sh --run-root ABSOLUTE_RUN_ROOT --stage STAGE --selection ABSOLUTE_SELECTION --source-manifest ABSOLUTE_SOURCE_MANIFEST --runtime-identity ABSOLUTE_RUNTIME_IDENTITY. It validates that run's manifests, source/runtime/aux identities, selection, scheduler IDs, terminal status, artifact records/owners, and selected artifact contracts; it never globs all run roots or submits or repairs workers.
3. Coordinate benchmark waves and reviewed lineage
- The exact wrapper must invoke the executor from the commit-keyed source
snapshot and carry the absolute source manifest, versioned runtime image
and runtime manifest, frozen
tree/auxroot, run ID, log root, and explicit selected-row scope. The durable manifest binds all of those source/runtime/aux identities; a later pull of the canonical checkout cannot change the wrapper betweenprepareand execution. serialization_groupis the required explicit cross-manifest mutex identity for a matching project, profile, and profile digest. Before any remote launch,launchtakes its deterministic group lock and scans sibling authoritative gate manifests for an activeRUNNINGgate with that identity. This bounded lock check is not a global run-root or artifact audit. Such a sibling records aresource_lock_conflictdiscrepancy, entersPRELAUNCH_STOP, and is not launched. Distinct groups may be prepared and launched concurrently only when their concrete artifact paths are disjoint, including gates that share a reviewed predecessor.- A top-level benchmark wrapper declares reviewed predecessor lineage through
repeatable absolute
--dependency-manifestpaths. Launch checks every predecessor for the same project, profile, and profile digest, stateCOMPLETED, a passing first audit, required reviewer approval, and a boundlocal_manifest, without requiring matchingserialization_group. Dependency lineage is independent of the mutex identity. - The exact command string in the manifest is immutable. Do not manually split a wrapper or rerun a wrapper after an ambiguous launch. New compute gates are not selected from old gate state; selection comes from the explicit run manifest and artifact contract.
- Existing remote evidence may be adopted only after the complete remote
manifest validates and binds every required identity/path field, including
serialization_group,dependency_manifests, source/runtime/auxiliary manifests, and the exact run identity. A tmux session must have exactly one pane across all windows whosepane_start_commandis the declared runner; process fallback parses exact argv rather than accepting a runner-path substring. Any mismatch is a fail-closed stop. - Before the one terminal
inspect, the completion task invokes the exact run-scopedecoda_run_audit.shcommand from Section 2 against the recorded run root. It must validate only the selected run's manifests, scheduler records, status, artifact records/owners, and selected contracts; it must not glob${HPC_SCRATCH_DIR}/_ecoda_runs/*, inspect mutable currentHEAD/Pixi fingerprints, submit workers, or repair compute.
4. Launch, wait, and inspect durably
- A bounded Luna Max task executes
reconcile → prepare → launchand exits after updating the checkpoint. Main arms exactly one harness-owned asynchronouswaitwith no periodic update requests. - The runner uses
/bin/bashon Yggdrasil, starts in the declaredremote_workdir, and changes to that directory before invoking the exact wrapper. It writes terminal status atomically and signals completion only after a durable primary or fallback failure status exists. Every terminal status must carry the current manifestevent_generation; stale generations are rejected. Monitoring files/events may be repaired only when that cannot duplicate or disturb the wrapper; stale generation markers are ignored. - A fresh bounded Luna Max task reads the durable wrapper log, extracts
every scheduler ID emitted after launch, and runs the first
inspectexactly once with repeatable--scheduler-array-id/--scheduler-watchdog-idoptions. The ECODA profile setsrequire_scheduler_ids: true;inspectfails closed before any accounting or audit command when neither an array nor watchdog ID was recorded or supplied. Otherwise it atomically recordsaudit_state: IN_PROGRESSbefore issuing any query, then performs the only terminal scheduler accounting operation: one strictsacct -n -P -X -j ... --format=JobIDRaw,State,ExitCodequery for all recorded scheduler IDs. Every ID needs an exact or array-taskJobIDRaw|State|ExitCoderow and every matched row must beCOMPLETED|0:0; missing, partial, pending, failed, malformed, or nonzero rows fail the audit. The final evidence write atomically recordsaudit_state: COMPLETED; an interrupted retry fails closed onIN_PROGRESSwithout repeating accounting or audits. Never pollsqueue/sacctor use an event as evidence. - A Luna Max reviewer invokes the second
inspect --approve-reviewerphase. That phase records approval without rerunning audit/accounting. Only a reviewedCOMPLETEDpredecessor releases a dependent gate; dependency lineage is independent ofserialization_group. Before remote launch, the deterministic same-group lock rejects an activeRUNNINGsibling with the same project/profile/profile-digest/group, recordsresource_lock_conflict, and moves this gate toPRELAUNCH_STOP. Distinct groups may proceed in parallel, including gates sharing a reviewed predecessor.
5. Preserve scientific and artifact contracts
- Biological labels (
Status,sample.origin,cond,Disease_Identity, and equivalent ground truth) are never preprocessing, HVG, normalization, batch-correction, embedding, or model covariates. - Keep the established DESeq2 modes: default
blind=TRUE, no batch column, no batch correction; batch-effect mode is explicitly batch-only withblind=FALSE,batch_col=<batch>, andcorrect_batch=TRUE. - Preserve raw counts in
layers["counts"], sample identity/order in Feather, and features/labels/distances in atomic RDS result bundles. Reject missing, NA, or mismatched identifiers. Verify the canonical NAS${NAS_TARGET_DIR}/benchmark/checksums.md5is nonempty (test -s) before runningmd5sum -cor consuming RDS. - Missing status, failed workers, exhausted OOM retry, incomplete accounting, checksum mismatch, or malformed wrapper-produced artifacts is a failure, not an optional warning.
- Gate-specific artifact shape, identifier, and provenance checks are defined by the authoritative execution plan and performed by the completion agent against artifacts the wrapper actually produced; do not invent generic provenance or source-manifest files.
- Use the
_debugJoanito five-sample subset for routine verification. Do not run full cohorts for minor checks.
6. Apply fail-closed recovery
- An ambiguous launch becomes
PRELAUNCH_STOP: preserve all evidence and reconcile before any retry. Never guess that the wrapper did not start. - Repair and reuse begin with a validator-only preflight of the explicitly
requested artifact rows. Reuse is allowed only when each selected artifact
satisfies its nonempty/schema/identity/checksum contract. If every row is
valid, write
NOOP_VALIDATEDto the specified run report and do not call durable-gateprepareorlaunch; no scheduler IDs are created. If any row is missing or invalid, submit only those explicitly scoped rows through the durable gate. Never broad-force a pipeline or infer scope from gate history. - An old
FAILED,PRELAUNCH_STOP, or stale gate is evidence only and cannot create a new scheduler selection.--forceremains row-scoped and requires an explicit dependency or integrity reason. A legacy run without its run-bound source/runtime manifests exits withlegacy_source_unpinnedbefore any worker, retry, or scheduler submission; validator-only inspection is the sole allowed path for such a run. - Every compute or retry path must carry the exact immutable source snapshot,
source manifest, frozen
tree/auxidentity, runtime identity, run ID, and selected-row scope. Run-scopedecoda_run_audit.shvalidates those manifests, status, scheduler IDs, artifact records/owners, and selected contracts before terminal inspect. Recovery never performs a global_ecoda_runs/*scan or couples a new run to mutableHEAD/Pixi state. - A nonzero wrapper, unexpected terminal accounting state, unknown process,
path/config discrepancy, missing or mismatched source/runtime/aux manifest,
ownership conflict, or audit failure records
FAILED/STOP, starts no dependent gate, and leaves the checkpoint unresolved for review. - Never manually synchronize partial outputs after wrapper failure. Do not
delete logs, manifests, status, checksums, or discrepancy records. A later
retry is a new explicitly reconciled gate, not an ad-hoc
rsync, and repair remains no-compute by default.
Profile severity and audit contracts
references/profile.json is the machine-readable policy. The following
severity rules govern any discrepancy it reports:
- BLOCK: unknown discrepancy; exact-command or gate-identity mismatch;
unexpected tmux/process; missing or non-atomic status; wrapper nonzero;
non-terminal/non-
COMPLETEDaccounting; missing artifact; checksum, path, source-level label-leakage, or scientific-invariant failure; a failed gate-specific shape/identifier/provenance check defined by the authoritative plan and run by the completion agent; missing or mismatched run-bound source/runtime/auxiliary manifests; manual/partial synchronization. Do not launch a dependent gate. - APPROVAL_REQUIRED: a discrepancy whose
kindis explicitly listed inpolicy.known_discrepancies. Preserve the raw evidence and obtain explicit user approval before proceeding; approval does not erase or downgrade the recorded discrepancy. - PASS: all exact gate identity, status, accounting, configured-root,
checksum, run-bound source/runtime/auxiliary manifest, and plan-defined
scientific/artifact audits succeed. Only a reviewed
COMPLETEDresult is releasable.
Audit contracts are evaluated only at the terminal inspect step:
- gate/manifest identity and exact immutable wrapper command;
- atomic terminal status and completion signal;
- one strict
sacct -n -P -X -j ... --format=JobIDRaw,State,ExitCodequery covering every recorded scheduler ID; - canonical repository/configured scratch/NAS roots and required wrapper presence;
- the exact run-scoped validator invocation
ecoda_run_audit.sh --run-root ABSOLUTE_RUN_ROOT --stage STAGE --selection ABSOLUTE_SELECTION --source-manifest ABSOLUTE_SOURCE_MANIFEST --runtime-identity ABSOLUTE_RUNTIME_IDENTITY, validating that run's source/archive, runtime, frozentree/aux, selection, scheduler-ID manifest, terminal status, artifact records/owners, and selected artifact contracts; - nonempty canonical NAS checksums and any plan-defined artifact shape, identifier, and provenance checks against artifacts the wrapper actually produced; and
- the authoritative plan checkpoint and Luna Max reviewer evidence.
The run-bound source snapshot and stage runtime.identity are the provenance
authorities. profile.json intentionally has "immutable_fingerprints": []
for snapshot-backed runs: do not fingerprint mutable canonical-checkout
git HEAD, live pixi.toml, or live pixi.lock as a terminal gate
contract. Do not glob ${HPC_SCRATCH_DIR}/_ecoda_runs/* or require broad
stage-neutral run-root/status scans; ecoda_run_audit.sh validates only the
recorded run.
Gate-specific artifact shape, identifier, and provenance checks are
intentionally not broad profile contracts. The authoritative execution plan
must name the existing wrapper-produced artifacts and expected checks, and
the completion agent must run and record those checks against those artifacts.
Require the exact run-bound source.manifest and runtime.identity copies
where specified by the run contract; do not invent generic provenance.json
or unrelated global source-manifest files.
The source-level no-label-leakage rule remains mandatory (see Section 5). This profile does not scan benchmark output strings for biological labels: ground-truth labels are legitimate result content, and runtime output strings cannot establish which covariates a model received.
The event is a wake-up mechanism, not proof. If transport is lost, reconnect and read durable status; do not relaunch or poll.
Rate Limiting (if applicable)
None. The profile uses SSH, tmux, filesystem state, and one terminal sacct
query; it calls no external HTTP API. Scheduler accounting is lifecycle-limited
and must never be converted into a polling loop.
Common Mistakes
- Running the wrapper from
$HOME/scratch/ECODA_paper, a login node, or a second tmux session instead of the Yggdrasil git clone and explicit gate session. - Editing the Yggdrasil checkout directly instead of making the change locally, committing and pushing it, and pulling the exact revision on Yggdrasil.
- Using bare interpreters, mutating
datasets.json/the lockfile, passing labels into a method, or skippingsrc/slurm_config.sh. - Relaunching after an SSH drop, treating event delivery as completion, or manually rsyncing partial outputs after a failure.
- Polling while a subagent remains alive, running more than one terminal
sacct, releasing a dependent gate before inspect and Luna Max review, or treatingserialization_groupas dependency lineage rather than the required mutex identity, or ignoring aresource_lock_conflictPRELAUNCH_STOP.