ai-monitor

Monitor a live AI feature — what to log, the signals that call for a look, a monthly error analysis, a re-evaluation on every prompt or model change, and cost and latency against budget. Use when the

SKILL.md

GitHub

Monitor

Set up how the team will know when the live AI feature in "$ARGUMENTS" gets worse, and run the regular review.

Arguments. $ARGUMENTS may contain the project name and optional flags. Take the first token as <project> and treat --skip-checks as a flag; never include flags in file paths. All paths below use <project>. If --skip-checks is present, skip the prerequisite check and, before writing any artifact, append one line to .singularskills/log.jsonl: {"ts":"<ISO timestamp>","skill":"ai-monitor","workflow":"ai","phase":8,"project":"<project>","event":"skipped","artifact":"08-monitor.md","note":"<reason or \"none given\">"}.

Compute with code. Perform every calculation (totals, balances, ratios, variances, projections) with a script — Bash, Python or a spreadsheet formula written to a file — and include the computation alongside the result. Never do arithmetic mentally.

Prerequisites

Part of a workflow. This skill is phase 8 (Monitor) of the AI Feature Lifecycle workflow (ai); cross-domain workflows run it too. First check .singularskills/log.jsonl for an open run for this project: a run-started line with "project":"<project>" and no later run-finished line for the same workflow and project. If one exists, the conductor is running you: continue here. Otherwise, if the user asked to start, run, continue or resume a process or workflow rather than this one step, do not continue here: invoke the workflow skill with the workflow the user asked for, or ai when they named none, followed by <project>; it runs this phase in order, asks before it writes and records the run. Otherwise continue here: a later phase runs on its own, behind its gate.

Run this skill's gate before writing: sh <this skill's folder>/scripts/gate.sh <project> <project root>, where <project root> is the folder that contains .singularskills/ (use . when you are already there). If it exits non-zero, stop and tell the user what it printed, unless --skip-checks was given (see Arguments). On a refusal, before you stop, append this line to .singularskills/log.jsonl (create the folder and file if missing): {"ts":"<ISO timestamp>","skill":"ai-monitor","workflow":"ai","phase":8,"project":"<project>","event":"refused","artifact":"08-monitor.md","note":"<what the gate printed>"}. Once the gate has passed, or under --skip-checks once you have appended the skipped line, append this line before you start work: {"ts":"<ISO timestamp>","skill":"ai-monitor","workflow":"ai","phase":8,"project":"<project>","event":"started","artifact":"08-monitor.md","note":""}.

Read .singularskills/ai/<project>/07-ship.md for the launch bar, human review and rollout. Read .singularskills/ai/<project>/06-evaluate.md for the checks.

If .singularskills/ai/<project>/08-monitor.md already exists, keep it as 08-monitor-<date>.md (today's date as YYYY-MM-DD; if that name is taken, add -2, -3 and so on) before you write, and compare against it.

Process

  1. What to log, per request: the input, prompt version, model, retrieved sources, output, latency, cost and feedback, with personal data handled as 05-build.md specifies.

  2. Signals that call for a look: negative feedback above a rate, handovers to a person above a rate, a check's pass rate below its bar, latency or cost above budget, and any output a hard check blocked. Each with its threshold (target) and who looks.

  3. The regular review (monthly, or as the user sets): a fresh error analysis on a sample of recent outputs, as /ai-evaluate runs it, looking for new failure groups. When the user supplies recent outputs or logs now, run it: compute the signals and the sample's notes and groups with a script written to .singularskills/ai/<project>/. Scripts read the team's outputs and logs where they keep them and write aggregates under .singularskills/; a quote from an output names its row or output ID, and names, emails, phone numbers and account IDs in it are replaced with placeholders.

  4. Re-evaluate on every change: any prompt version or model change re-runs the checks before it ships; say this is a rule, and who enforces it.

  5. Cost and latency: actual against budget, computed from the logs supplied or not measured.

  6. Write the artifact to .singularskills/ai/<project>/08-monitor.md with sections Logging, Signals, This review (or "first review not yet run"), Change rule, and Cost and latency.

Output

The monitoring plan and review written to .singularskills/ai/<project>/08-monitor.md. Present a summary to the user highlighting:

  • The signals and their thresholds
  • Any signal over its threshold now
  • New failure groups since the last review

Record the run. After you have written the artifact, append this line to .singularskills/log.jsonl (the started line of Prerequisites, with "event":"finished"): {"ts":"<ISO timestamp>","skill":"ai-monitor","workflow":"ai","phase":8,"project":"<project>","event":"finished","artifact":"08-monitor.md","note":""}.

Open the artifact with this frontmatter (agent is the agent you run in, or unknown; gate is passed when the gate ran clean and skipped under --skip-checks):

---
skill: ai-monitor
workflow: ai
phase: 8
project: <project>
started: <ISO timestamp>
finished: <ISO timestamp>
agent: <the agent you run in, or "unknown">
gate: <passed|skipped>
---

End the artifact with ## Decisions and ## Open questions, in this shape:

## Decisions
- <what was chosen> — <why> — <what was rejected>

## Open questions
- <what the next phase or a reviewer must settle>