monitor-jobs
Monitor SLURM/PBS jobs and their logs. Use when the user asks to monitor, check, or watch a submitted job. Automatically detects running/pending jobs, tails logs, and reports errors.
SKILL.mdskills/monitor-jobs/SKILL.md
GitHubMonitor HPC Jobs
You are monitoring HPC jobs on a cluster. Follow this workflow:
Step 0: Resolve project root and environment
- Look for
.envby walking up from$PWD(check$PWD/.env,$PWD/../.env, etc.) - The directory containing
.envis$PROJECT_ROOT - If no
.envis found, fall back to$PWD - Read
.envto getJOBS_DIR(used for locating job scripts and logs).
Step 1: Detect the scheduler and cluster
command -v sbatch && echo "SLURM" || (command -v qsub && echo "PBS" || echo "UNKNOWN")
Then identify which cluster you are on:
# CX3 (Imperial) — PBS Pro
[[ "$PWD" == /rds/general/user/* ]] && echo "CX3"
# Isambard — SLURM
[[ "$PWD" == /home/*/* ]] && ls /projects/ &>/dev/null && echo "ISAMBARD"
If on CX3, also check if you are on a login node (hostname contains login). If you are NOT on a login node, you can run short Python commands directly without submitting a job.
Follow the SLURM or PBS Pro paths below depending on the scheduler.
Step 2: Identify the job
- If
$ARGUMENTScontains a job ID, use that.
SLURM
- Run
squeue -u $USER --format="%.18i %.20j %.8T %.10M %.6D %.4C %.20R %o" --sort=-Vto find the user's active jobs. - If no jobs are running/pending, check recently completed jobs with
sacct -u $USER --starttime=$(date -d '24 hours ago' +%Y-%m-%d) --format=JobID,JobName%30,State,ExitCode,Start,End,Elapsed --noheader.
PBS Pro (CX3)
- Run
qstat -u $USERto find the user's active jobs. - If no jobs are running/queued, try
qstat -f <job_id>if a job ID was provided (may still work for recently-finished jobs). Otherwise, report that no active jobs were found — PBS Pro has nosacct-equivalent history command.
Step 3: Get job details
SLURM
- Run
scontrol show job <job_id>to get full job info (WorkDir, StdOut path, submission script, state, node allocation). - Extract the log file path from the
StdOutfield. Note:%jin the path is replaced by the job ID.
PBS Pro (CX3)
- Run
qstat -f <job_id>to get full job info (Output_Path, Error_Path, job state, resources). - Extract both log paths:
Output_Path(stdout) andError_Path(stderr). PBS Pro uses separate files unlike SLURM. - PBS log filenames are fixed strings — there is no
%jjob ID expansion. - Log paths from
#PBS -o/#PBS -eare relative toPBS_O_WORKDIR(the submission directory). Resolve them accordingly.
Step 4: Wait for job to start (if PENDING/QUEUED)
If the job is PENDING (SLURM) or Q (PBS), do NOT just report and stop. Instead, automatically poll until it starts:
- Report the current state: "Job 12345 is PENDING. Waiting for it to start..."
- Run
sleep 30in the background usingrun_in_background: true. - After each sleep, re-check:
- SLURM:
squeue -j <job_id> --noheader --format="%.18i %.8T" - PBS:
qstat <job_id>(check the state column)
- SLURM:
- If still pending, report briefly ("Still pending, waited Xm so far...") and sleep again.
- Repeat until the job transitions to RUNNING (R) or FAILED/CANCELLED.
- Once RUNNING, wait 10 more seconds for the log file to appear, then proceed to Step 5.
This polling loop is the default behavior — do not ask the user whether to wait. Just do it.
Step 5: Monitor the logs
- Check if the log file exists yet. If not, sleep 10 seconds and retry (up to 5 times).
- Once the log file exists, read the last 100 lines using
tail -n 100 <log_path>. - PBS Pro: Check both the
.out(stdout) and.err(stderr) log files. Errors often appear only in the.errfile. - Look for:
- Errors: Python tracebacks,
Error,Exception,FAILED,CANCELLED,OOM,CUDA error,RuntimeError - Warnings:
UserWarning,FutureWarning, deprecation notices - Progress indicators: epoch numbers, step counts, loss values, W&B run URLs
- Training metrics:
train/loss,val/loss, learning rate, gradient norms
- Errors: Python tracebacks,
Step 6: Report status
Provide a concise summary:
- Job state (PENDING / RUNNING / COMPLETED / FAILED / TIMEOUT)
- Runtime so far
- Training progress (current epoch/step, latest loss values)
- W&B run URL if visible in logs
- Errors or warnings if any — include the full traceback
Step 7: If errors are found
- Analyze the error and identify the root cause.
- Check the relevant source files in the codebase to understand the issue.
- Propose a fix. If the fix is clear and safe:
- Apply the fix to the source code.
- Cancel the failed job (ask user first):
- SLURM:
scancel <job_id> - PBS:
qdel <job_id>
- SLURM:
- Resubmit the job with the original script.
- If the fix is ambiguous or risky, present options to the user before acting.
Step 8: Continuous monitoring (if job is still running)
If the job is running and healthy, offer to keep monitoring. If the user wants continued monitoring:
- Sleep for 60 seconds between checks.
- Re-read the last 100 lines of the log.
- Report any new errors, warnings, or progress updates.
- Stop monitoring when the job completes, fails, or the user asks to stop.
Key paths
| What | Path |
|---|---|
| Job scripts | $JOBS_DIR/<project>/[<group>/] (the optional <group>/ sub-dir is used by /softnano:submit-wandb-job) |
| SLURM logs | $JOBS_DIR/<project>/[<group>/]logs/ (or as specified by #SBATCH --output) |
| PBS logs | $JOBS_DIR/<project>/[<group>/]logs/ (or as specified by #PBS -o / #PBS -e) |
| Env vars | $PROJECT_ROOT/.env → JOBS_DIR |
CX3 (PBS Pro) Reference
Queue Reference
| Queue | Walltime | CPUs | Memory | GPUs | Use case |
|---|---|---|---|---|---|
v1_gpu72 |
72h | 1–64 | up to 920 GB | L40S / A100 | GPU inference/training |
v1_medium24 |
24h | 1–64 | up to 450 GB | — | CPU-only, moderate memory |
v1_largemem72 |
72h | 1–128 | 921–4000 GB | — | mmseqs2 / large-index jobs |
12 GPU limit per user on
v1_gpu72.
CX3 Storage
| Path | Size | Notes |
|---|---|---|
/rds/general/user/<user>/home |
1 TB | Code, logs, outputs |
/rds/general/user/<user>/ephemeral |
10 TB | Large datasets (auto-deleted every 30 days) |
Key Differences from SLURM
- No
srun— run Python directly after activating the venv. - No
%jjob ID expansion in#PBS -o— use fixed log filenames. - Resource syntax:
select=<N>:ncpus=<C>:mem=<M>gb:ngpus=<G>instead of--nodes/--gpus. - Variables via
qsub -v "KEY=val"instead of--export. - Queue via
#PBS -qinstead of#SBATCH --partition.
Important notes
- SLURM: Always resolve
%jin log paths to the actual job ID. - PBS Pro: Monitor both
.outand.errfiles. Log filenames are fixed (no job ID expansion). - If the log path is relative, resolve it relative to the job's WorkDir (SLURM: from
scontrol, PBS: fromPBS_O_WORKDIR). - When sleeping/waiting, use
sleepin Bash withrun_in_backgroundto avoid blocking. - Present errors with enough context (surrounding lines) for the user to understand.
- See the
cluster-instructionsskill (isambard.mdfor SLURM,cx3.mdfor PBS Pro) for full HPC reference.