kortix-harness-refinement
The continual-harness refinement protocol — how an agent inspects its recent trajectory for failure signatures and improves its own harness (agent prompts, sub-agents, skills, memory) in place, mid-se
SKILL.mdpackages/starter/templates/managed/skills/kortix-harness-refinement/SKILL.md
GitHub<skill name="kortix-harness-refinement">
<overview>
Your **harness** is everything in this repo that shapes how agents work
here. It has five components, all in git:
| Component | Location | What it is |
|---|---|---|
| Prompts | agents/<name>.md, named by agents.<name>.prompt_file in kortix.yaml (or inline agents.<name>.prompt) |
Each agent's instructions and strategy |
| Sub-agents | agents.<name> blocks with mode: subagent; an agent starts only the ones in its agents.<name>.subagents |
Specialist agents the orchestrator invokes |
| Skills | skills/<name>/SKILL.md, registered under skills:; an agent loads only the ones in its agents.<name>.skills |
Reusable routines: text heuristics and guides |
| Tools | tools/<name>.ts, registered under tools:; pi packages under extensions:, enabled by agents.<name>.extensions |
Executable code: wrappers, scripts, automations |
| Memory | memory/ |
Persistent facts, strategies, observations |
An agent's model, permission and grants live in its agents.<name> block
in kortix.yaml.
Refinement means: read your recent trajectory, find where the harness
failed you, and fix the harness — not just the immediate task. A memory
edit is readable on your next turn: memory is plain files. An edit to an
agent prompt, skill, or tool reaches sessions once its change request
merges; a running session loads it with kortix sessions reload <session id>.
You do not restart, and the value compounds over the life of the workspace.
This protocol has two operating modes:
- In-session refinement (self-invoked) — YOU run it, inside your own working session, the moment a failure signature costs you twice. Also run a checkpoint on long sessions: after every ~25 turns of work, pause and scan your recent turns before continuing. You refine based on YOUR OWN trajectory, then resume the task.
- Workspace-level reflection (optional) — a new workspace has no
reflection agent and no reflection trigger. A workspace may add one: an
agent (for example
agents/harness-reflector.md) that a cron trigger inkortix.yamlruns. It fans out read-only reviewer subagents to work through every recent session's full history, aggregates their findings, and refines the shared harness onmainvia a change request. Every run creates a sandbox. This skill is its protocol: the agent's prompt only needs to load it.</overview>
- Repeated tool/command failures — the same command or tool errors more than once, or you retried a broken approach. → Repair the tool, or record the working alternative in a skill.
- Rediscovery loops — you (or another session) re-derived something a past session already knew (an API quirk, a file location, a decision). → Memory entry.
- Stalled objectives — turns pass without progress toward the stated goal; you circled, re-read, or re-planned without acting. → Prompt guidance for the responsible agent, or a decomposition sub-agent.
- Repeated multi-step patterns — you performed the same 3+ step sequence more than twice by hand. → Codify it: a skill (if guidance) or a tool (if executable).
- Exception-raising code — an executable tool or script (a
tools:module or anextensions:package inkortix.yaml) raised; you worked around it instead of fixing it. → Repair the code now. - Missed opportunities — information or shortcuts visible in the trajectory that no component captured. → Whichever component fits.</failure-signatures>
Pass 1 — Prompts (Δp). Reread the responsible agent's prompt file against the identified failures. Tighten instructions that were ignored, add the missing rule, remove guidance that no longer earns its tokens. Keep prompts short; a prompt that only grows is a failing prompt.
Pass 2 — Sub-agents (ΔG). Create a sub-agent only for a pattern that
recurred across the window AND needs its own scoped permissions or
prompt: an agents.<name> block with mode: subagent, added to the
orchestrator's subagents list. Edit sub-agents implicated in failures.
Delete sub-agents that have not been invoked productively — check before
keeping.
Pass 3 — Skills and tools (ΔK). Codify successful sequences from the
trajectory into a skill (guidance) or a tool (executable). Repair every
tool the trajectory shows raising exceptions. Prefer editing an existing
skill over creating a near-duplicate. A new skill is a folder named after
it (skills/<name>/SKILL.md), registered under skills: and added to the
skills grant of each agent that needs it. A new tool is registered under
tools: and reaches each agent whose tools grant includes it. An
unregistered file reaches no agent.
Pass 4 — Memory (ΔM). Follow the kortix-memory skill's rubric with
the file tools on memory/: fill gaps the trajectory exposed, update stale
entries, demote or delete entries about areas the workspace has moved past.
Keep MEMORY.md in sync.
Scope discipline: fix what the trajectory shows. Do not speculatively rewrite components with no observed failure. Most refinement runs should touch one or two components, not all five. </four-passes>
<project-review-fanout> This section applies only to a top-level reflection run. If you loaded this skill as a subagent, skip the fan-out and review the sessions yourself.For workspace-level reflection, do not skim a digest and call it a review. Work through every session:
- Enumerate every session in the window:
kortix sessions digest --since 24h --json(add--allfor a first ever run). Note per session: id, agent, status, title, whether a live transcript is available. - Fan out one read-only reviewer subagent per session with the
agenttool (at most 4 at a time; review every session, skip none silently). Each reviewer gets the session id and returns a structured findings report. Reviewers gather the FULL picture for their session:- the transcript:
kortix sessions log <id> --json(a stopped session reads its saved transcript), - the session's git branch: its commits, diffs, and files touched,
- change requests the session opened and their review outcomes.
- the transcript:
- Aggregate all reviewer reports. Deduplicate findings that recur across sessions — a failure signature seen in three sessions outranks one seen once. Rank by cost (turns wasted × sessions affected).
- Run the four passes on the ranked findings, then land per the rules below.
Sub-agent review is read-only: reviewers never edit the harness or open CRs. Only the orchestrating reflection agent writes. </project-review-fanout>
<landing-rules> **In-session refinement (session branch — immediate):**Apply the edits in your clone of the repository (
kortix workspaces clone;/workspacehas none at boot). They do not change the config this session runs: agents, skills and tools compile from the base branch. They take effect once the change request merges, in new sessions and in running ones when they are idle (kortix sessions reload <session id>applies it now).Commit them separately from task work:
# Stages the harness files this project has. Add any other harness path you edited. git add -A -- $(ls -d kortix.yaml agents skills tools extensions commands memory 2>/dev/null) git commit -m "harness: <one-line summary of what failed and what changed>"Push and open (or update) ONE change request per session for harness promotion to
main:git push origin HEAD kortix cr open --title "harness: <summary>" \ --description "Failure signatures observed, edits per component, evidence (commands/turns)."If this session already has an open
harness:CR, push to it instead of opening a second one.Return to the task. A refinement interruption ends with you resuming what you were doing, with the improved harness in effect.
Workspace-level reflection: all edits land only
via a CR against main. Nothing applies immediately; the merged CR is
what future sessions inherit.
</landing-rules>