training-check

Monitors running spatial experiments. Checks output files, log files, and process status. Categorizes results as OK, STALLED, FAILED, or COMPLETE. Fires alerts by appending to output/PROJ_NOTES.md. Ru

SKILL.md

GitHub

Skill: training-check

You monitor spatial experiment execution and detect problems early to avoid wasting compute time.


Phase 1: Check Active Experiments

Read output/EXPERIMENT_LOG.md for experiments with status RUNNING or PENDING.

For each running experiment:

  1. Check if output file was recently modified:
python -c "import os, time; f='[output_path]'; age=(time.time()-os.path.getmtime(f))/60; print(f'Modified {age:.1f} min ago')"
  1. Check last few lines of log file for errors:
tail -20 [log_path]
  1. Check for error keywords: Error, Traceback, NaN, inf, MemoryError, Killed
  2. Check for stall: no file modification in > 30 min despite RUNNING status

Phase 2: Classify Status

Signal Classification Action
Output file updated in last 15 min, no errors OK Continue monitoring
Log contains "NaN" or diverging loss CLEARLY BAD Terminate, log failure
Output file not modified in > 30 min STALLED Investigate process
Results file written, contains valid metrics COMPLETE Update EXPERIMENT_LOG
MemoryError in log MEMORY FAIL Retry with smaller sample
Process not running, no output file DEAD Re-queue if < 2 retries

Phase 3: Actions

COMPLETE: Update output/EXPERIMENT_LOG.md status to SUCCESS. Append finding to output/PROJ_NOTES.md. CLEARLY BAD: Kill job if possible. Mark as FAILED in log. Try to fix the issue and retry once. STALLED: Check if process is still alive. If dead: re-queue. If alive: wait 15 more min.


Phase 3.5: Human Checkpoint — Data Synthesis

Honor the HUMAN_CHECKPOINT flag in CLAUDE.md (default: true). This is a monitoring skill, but a few automatic recovery actions silently produce or substitute experiment data. When HUMAN_CHECKPOINT is true, PAUSE and request explicit user approval before any of the following; when false, log the action to output/PROJ_NOTES.md with rationale and proceed.

Trigger Show before pausing
MEMORY FAIL → retry with smaller sample: about to subsample, downsample, or otherwise reduce the dataset to make the run fit Original N, proposed N, sampling rule (random / stratified / spatial), seed, and the claim affected by the reduced sample
CLEARLY BAD → fix and retry: about to modify the run (reduce learning rate, drop a feature, change loss, swap optimizer) and re-launch Diagnosed cause, the exact code/config change, and confirmation that the new run is still valid evidence for the original claim
DEAD → re-queue: about to re-launch a run whose previous outputs were partial, with carry-over from a checkpoint or warm-start Which artifacts will be reused, which will be regenerated, and the risk of mixing pre-/post-fix data in the same result file
COMPLETE → mark SUCCESS: about to write a SUCCESS row using metrics that were imputed / interpolated across missing log lines, or backfilled from a different run Which numbers were imputed, source, and whether the run should be re-executed instead

Do not mark a run SUCCESS in EXPERIMENT_LOG.md based on synthesized metrics. If the user approves, append Synthesis approved: <action> — <user reason> — <date> to the run's notes.


Phase 4: Progress Report

Output to stdout:

Training Check — <timestamp>
Active experiments: N
  COMPLETE: N
  OK (running): N
  STALLED: N
  FAILED: N

[List any failures or alerts]

Append alerts to output/PROJ_NOTES.md: [ALERT] Experiment <name> failed: <reason>