nw-agent-evals
Lightweight eval method for testing nWave AGENTS and SKILLS (LLM behavior) as a lean alternative to heavy BDD/ATD. An eval = one prompt -> one captured run (trace + artifacts) -> a small set of checks
SKILL.mdnWave/skills/nw-agent-evals/SKILL.md
GitHubAgent Evals
Why this exists
Agents are LLMs — non-deterministic. Traditional BDD/unit testing tests them badly and breeds bloat (measured: 294K test LOC, 5.5:1 test:src, step reuse 1.10x vs >=4x target). Evals are the lean way to verify agent behavior: a few targeted signals instead of a monolithic spec.
This skill is the antithesis of ATD over-specification. Keep it small. The anti-pattern is up-front exhaustive specification — that IS the bloat.
nw-agent-testing (sibling skill) |
nw-agent-evals (this skill) |
|
|---|---|---|
| Form | static 5-layer manual checklist | executable dataset + grader + score over time |
| Use | one-shot design review of a spec | repeatable regression net for behavior |
| Output | pass/fail judgement | comparable score, trend across runs |
Use both: nw-agent-testing to vet the spec, nw-agent-evals to watch behavior over time.
What an eval is
One eval = prompt -> run -> checks -> score.
- prompt — a single input that should (or should NOT) trigger the agent/skill.
- run — one provider turn through the selected nWave adapter, with its available native evidence and artifacts captured.
- checks — a small set of targeted assertions (not one monolithic check).
- score — a comparable number you can track across runs to catch regressions.
Replaces "vibes" with measurable signals: did it invoke the right skill, run the expected tools, respect the conventions and produce the required observable effect? Never grade a terminal-text grammar as a behavioral outcome.
Definition of Done — before you write the eval
Write the success criteria FIRST, before implementing the agent/skill or its eval. Four check categories:
| Category | Question | Graded by |
|---|---|---|
| OUTCOME | Did the task complete through its required public effect and evidenced semantic result? | deterministic |
| PROCESS | Was required skill delivery or use evidenced, with only behavior-relevant tool observations? | deterministic |
| STYLE | Does the output respect nWave conventions (sections, format)? | model-graded |
| EFFICIENCY | No useless commands / no token blowup? | deterministic |
If you cannot state DoD before writing the skill, the skill's job is not yet defined — stop and define it.
Workflow
Run these steps in order:
- Define success first — write a small set of falsifiable checks in the four categories above.
- Use one representative probe when needed — capture the provider-native evidence that the selected adapter actually exposes.
- Add only the cases needed — include a positive case and a relevant negative control; grow the set from observed failures.
- Grade deterministic evidence — check public effects, admitted tool evidence, and structured artifacts. Do not derive a verdict from terminal prose.
- Grade semantic quality separately — use the existing rubric facility; retain narrative feedback as diagnostic evidence, not terminal control input.
Capturing provider evidence
The selected adapter owns the provider projection. Claude roles run through
the Claude adapter; Codex roles run through the Codex adapter's codex exec
projection. Do not substitute one provider's trace shape, tool names, or
launcher for the other's.
Distinguish these three facts when evaluating a skill:
| Fact | Evidence | What it establishes |
|---|---|---|
| Preloaded knowledge | The role declares the skill in frontmatter and load_role_instructions materializes it in the provider instruction. |
The role received the skill before the turn. It creates no native Read event. |
| Native read | A provider-native tool event reads a conditional SKILL.md path. Claude's existing transcript tracker records this form. |
The turn fetched that file on demand. It does not prove that the knowledge affected the result. |
| Actual use | A predicted, observable effect or a capability-specific tool/result is present, with no simpler explanation. | The role applied the skill sufficiently for this eval. A catalog entry, preload, or Read alone is insufficient. |
Evaluate semantic results and public effects separately from their presentation. Terminal prose, stdout, stderr, recorder entries and native tool traces carry diagnostic/provenance evidence, not interchangeable proof of a durable effect. Use the selected host's native transcript/tool receipts rather than another parser. Inspect only events and recorder evidence that its adapter preserves. If the required observation is absent on the selected provider, report that check as INDETERMINATE; do not infer it from another projection.
Check an artifact on disk as well as its reported tool event. Only ask for a tool when the behavior requires it. Tool order and an exact command sequence are not general quality signals.
Deterministic graders (nWave-native signals)
Parse the trace, assert mechanically. nWave-specific, high-value signals:
| Signal | Assertion | Why it matters |
|---|---|---|
| Declared preload | resolved role instruction contains the declared skill once | verifies delivery of resident knowledge, not its use |
| Conditional read | native Read event for the resolved skill path, where the provider exposes one |
verifies on-demand fetch, not semantic use |
| Capability-specific behavior | required public effect or justified tool/result is present | tests actual application of the skill |
| Semantic outcome observed | the semantic result and required public effect are evidenced | separates judgment from presentation |
| Artifact content | the required document contains the accepted facts | checks durable outcome rather than a completion claim |
| Efficiency | compare only a measured, relevant excess such as duplicate calls | avoids arbitrary ceilings |
| Negative control | the unsupported behavior or unnecessary skill effect is absent | guards against over-eager invocation |
Bind to the code-fact CLI where useful: e.g. assert the agent invoked code-fact query.callers-of rather than relying on a catalog entry (catalogued != wired). Note that feature-level change-scope analysis has no stable CLI today — an eval must not demand a capability the production CLI does not expose, or it grades the tooling rather than the agent.
Qualitative grader (model-graded rubric)
For STYLE / design-quality / review-quality, use the existing rubric facility. Keep each check single-purpose: identity, pass/fail judgement and evidence-located notes; retain the overall judgement and comparable score in the harness-owned result. Models supply judgements, not serialized rubric templates or terminal grammars. The harness owns result structure and recording; unavailable recording capability is INDETERMINATE, not permission to ask a role to print a data object.
When behavior under eval supplies a DESIGN decision, evaluate two linked observations: the values satisfy the Request, and the resulting authority contains them. Preserve one current authority rather than a competing ADR or brief. Revisions preserve unaffected facts; unsuccessful changes must not silently replace accepted authority.
For an EXAMINE eval, preserve the source-blind boundary. Give the examiner the available observation packet, not source access; an absent observation remains INDETERMINATE rather than becoming an inferred pass or failure.
Dataset
Use the smallest CSV dataset that covers the observed positive behavior and a relevant negative control. Minimum columns:
id,prompt,should_trigger,expected_skill,expected_observation,notes
ev-01,"Describe design facts for X",true,nw-design-patterns,"decision satisfies Request; resulting authority contains its facts",explicit
ev-07,"Just fix this typo",false,,,"negative control - no design facts"
- Mix: explicit-invocation, implicit-from-description (does
descriptionalone trigger it?), contextual, and NEGATIVE-CONTROLS (should_trigger=false). name+descriptionare the PRIMARY invocation signal — implicit rows test exactly that.- Coverage grows from real failures, never speculatively.
Where evals live
tests/evals/<agent-or-skill-name>/
dataset.csv # the prompt set
rubric.json # model-graded rubric (JSON-Schema)
README.md # DoD + how to run
runs/ # captured transcripts + scores per run (gitignored or pruned)
Reuse the existing tests/evals/ facility; do not create a new general
harness. Keep raw provider records out of committed bloat unless a test needs a
small, scrubbed fixture.
Principles
- Define success before you write the skill — no DoD, no skill.
- Small targeted checks beat monolithic ones — many cheap signals catch regressions early; one giant assertion hides them.
- Every manual fix is a future eval — coverage is earned from observed failures.
- Negative controls are mandatory — an agent that fires when it shouldn't is as broken as one that doesn't fire.
- name + description are the invocation contract — test them, don't bypass them with explicit invocation only.
- Least privilege — eval graders are read-only over traces + artifacts.
- Stay lean — over-specifying up front recreates the ATD bloat this method exists to avoid.
Example: eval for nw-solution-architect
DoD — the architect supplies valid semantic design facts and their resulting authority. When structural code facts are necessary and obtainable, they are provider-labelled and support the decision. A completion claim alone establishes neither valid content nor the durable effect.
Dataset rows (excerpt):
id,prompt,should_trigger,expected_skill,expected_observation,notes
sa-01,"Describe design facts for the handoff-state-algebra feature",true,nw-design-patterns,"decision satisfies Request; resulting ADR contains its facts",design
sa-02,"What ADRs exist for the gate layer?",true,,"provider-labelled code facts support the answer when code facts are needed",lookup
sa-03,"Rename this variable to camelCase",false,,,negative control - not an architecture task
Deterministic grader (over the captured trace + artifact):
- DELIVERY: the role's resolved instruction contains the declared preload once; a conditional native read is checked only when this case requires it.
- OUTCOME: the design facts satisfy the requested decision.
- OUTCOME: the declared ADR contains accepted facts, observed through the actual consumer surface.
- EVIDENCE: when structural analysis is necessary, its provider and confidence labels support the decision; do not demand a fixed shell spelling or order.
- NEGATIVE (sa-03): no architecture facts or durable-design production occurs.
Model-graded rubric (STYLE/quality):
| Check | Example judgement | Evidence note |
|---|---|---|
| Context states the problem | Pass | Clear problem framing in the generated ADR |
| Decision is singular | Pass | One owned decision addresses the supplied gap |
| Consequences quantify trade-offs | Fail | Trade-offs remain qualitative |
These are rubric judgements, not a role-output template. The harness constructs the stored result; terminal notes alone establish no recorded outcome.
Score = deterministic checks (binary, weighted) + rubric score, tracked per
comparable run. A missing required fact, generated document, or relevant supporting
observation flags a regression; unavailable evidence is reported as
INDETERMINATE.