opp-eval
Internal /ace:run step — Umbrella aggregator that rolls every per-skill -eval verdict into a run-level scorecard.
SKILL.mdskills/opp-eval/SKILL.md
GitHubOpportunity Eval
Aggregate every per-skill -eval verdict for an opportunity into a
single run-level scorecard, surface the weakest skills + dimensions,
and draft improvement recommendations. opp-eval is the umbrella
half of ACE's evaluation story — each per-skill eval judges its own
artifact in isolation; opp-eval judges the whole opp.
opp-eval is ad-hoc and opt-in. It is not part of the
--mode review auto-pause-at-gate flow. Run it anytime during or
after an opportunity to answer "how well did this run go overall, and
what should I improve?" — the operator's original ask for "one
overview judge/review agent that we can apply to overall runs."
See skills/README.md § QA vs Eval — the two-phase pattern for the
framework rationale and the uniform verdict-YAML shape that this
aggregator reads.
Inputs
Pipeline-only: rolls up the per-skill -eval verdicts of one run; its inputs are that run's verdict files (§ Process).
Modes
All paths below are run-scoped under
ACE/<opp-name>/runs/<run-id>/10-closeout/opp-eval/ (opp-eval's own
subfolder — it writes more than one artifact and per-run isolation
keeps re-runs clean).
| Mode | What it does | LLM cost | Writes |
|---|---|---|---|
--quick |
Structural-only: walk the artifact manifest for the opp's current phase, confirm every required (non-dated) artifact exists in the run folder. No verdict aggregation. | None | stdout summary + 10-closeout/opp-eval/opp-eval_scorecard-quick.md |
--deep |
Structural check plus aggregation: discover every runs/<run-id>/<phase>/*_verdict*.yaml, roll scores into skill-category dimensions, compute run-level verdict, emit per-skill narrative + improvement recommendations |
LLM-as-Judge (recommendation drafting) | 10-closeout/opp-eval/opp-eval_scorecard-deep.md (human) + 10-closeout/opp-eval/opp-eval_verdict-deep.yaml (machine) |
--monitor |
Same as --deep, plus append a one-liner to 10-closeout/opp-eval/opp-eval_trend.md |
LLM-as-Judge | Same as --deep + trend append |
If no mode is passed, default to --quick.
Process
Read the PDD for archetype context (best-effort, not required). Look up
ACE/<opp-name>/runs/<run-id>/1-design/idea-to-pdd.md. If present, read thearchetype:field and## Evidence Modelsection and keep them in scope for recommendation phrasing. If the PDD is missing, do not fail — opp-eval is designed to work on partially-completed opps (that's exactly the point of--quick). Emit[INFO] pdd.md not found — recommendations will be archetype-agnosticand continue.Read
run_state.yamlfromACE/<opp-name>/run_state.yamlto determine the opp's current phase. This drives which manifest entries to check in step 3. Ifrun_state.yamlis missing, assumedesignphase (only the earliest required artifacts are expected).Structural check (runs in every mode). For each required, non-dated artifact in
lib/artifact-manifest.tswhosephaseis at or before the opp's current phase, verify the path exists in the Drive folder viadrive_list_folder. Build three lists:present,missing,unexpected. Mirror the shape ofvalidateFixture()inlib/artifact-manifest.ts— opp-eval is effectively the live-folder equivalent of that fixture-validation helper.Short-circuit on
--quick. Write the quick scorecard (see § Quick-mode output below), print a one-line stdout summary (present/expected artifacts, N missing), and stop. Do not read any verdict files. No LLM calls.Discover verdicts. Walk each phase folder under
ACE/<opp-name>/runs/<run-id>/(1-design/,3-commcare/,4-connect/,5-ocs/,6-qa-and-training/,8-solicitation-management/,9-execution-manager/,10-closeout/) and collect every file matching*_verdict*.yaml. Also descend one level into10-closeout/opp-eval/for the umbrella's own verdicts (excluded from aggregation byskill: opp-evalfilter — see step 6). For each file, parse as YAML. Expected top-level keys perskills/README.md § QA vs Eval — the two-phase pattern:skill,target,mode,ran_at,capture_path,overall_score,verdict,dimensions, optionalper_item, optionalgate.Tolerate missing fields. If a verdict YAML is malformed or missing a required key, do not crash. Emit
[INFO] skill <X> verdict missing required field <Y>and continue. The aggregator must surface gaps, not hide behind them.Group verdicts into skill categories. Map each verdict's
skillto one of seven run-level dimensions:Category Weight Covers skills design0.20 idea-to-pdd,pdd-to-test-promptscommcare0.18 pdd-to-learn-app,pdd-to-deliver-app,app-deploy,app-test-cases,training-llo-guide,training-flw-guide,training-quick-reference,training-faq,training-onboarding-email,training-deck-generateconnect0.13 connect-program-setup,connect-opp-setupocs0.18 ocs-agent-setup,ocs-chatbot-eval-quick,ocs-chatbot-eval-deepsolicitation0.10 solicitation-create,solicitation-review(added 0.12.0 — Phase 8)operate0.13 llo-onboarding,llo-uat,llo-launch,ocs-chatbot-eval-monitor,flw-data-review,timeline-monitorcloseout0.08 opp-closeout,llo-feedback,learnings-summary,cycle-gradeCategory weights sum to 1.0. Per-category score is the simple mean of all verdicts that map into the category, excluding any verdict with
verdict: incomplete(added 0.9.8). Anincompleteverdict means the rubric correctly detected a structural gap (degraded mode, phase not run, target artifact missing) and refused to grade — that's a coverage gap, not a quality signal, and shouldn't affect the category mean.Categories with zero gradable verdicts (zero non-incomplete) score
nulland their weight is redistributed proportionally across categories that did score — so a partial opp (e.g., still in Phase 4, no OCS verdict yet) gets a meaningful run-level number rather than an artificially low one.Compute the run-level overall score. Weighted mean of the non-null category scores, with weights renormalized per step 6. The formula is:
sum(score_i * weight_i) / sum(weight_i)for categories wherescore_iis not null.Classify the run-level verdict (coverage-aware as of 0.8.0). The raw weighted-mean score is computed first; then coverage caps the verdict regardless of score. This is the explicit fix for the "1/6 categories scored 8.9 → run-level PASS" inflation surfaced by the first smoke run.
Coverage tier (count of categories with score ≠ null — i.e., categories with at least one non-incomplete verdict — ignoring categories with
expected_at_phaselater than the opp's current phase):Categories scored Tier Verdict cap Notes 0 none incompleteNothing to grade 1 thin incompleteSingle-category score isn't a run grade — flag [INFO]2 partial warn(max)One cross-skill signal exists but most of the run is unmeasured 3 adequate passif raw ≥ 7Less than half-coverage; raw score governs 4 adequate passif raw ≥ 7Half-coverage; raw score governs 5+ full passif raw ≥ 7;warn4.0–6.9;fail<4.0Cross-skill view is real The verdict cap applies AFTER the raw score: a 9.5 raw with 1-category coverage emits
incomplete, not a misleading PASS.incompleteis a new verdict introduced in 0.8.0 — downstream readers treat it as "needs more rubric coverage before this number is meaningful," not as a quality fail.Deployability cap (added 2026-05-29): the verdict is ALSO capped at
warnby the independent deployability spot-check (step 10b) when that probe scores < 5 against a would-be PASS rollup. This is the anti-laundering backstop — coverage breadth is not fitness depth, and a high rollup of inflated per-skill scores must not survive an independent raw-artifact judgment that the build is undeployable.Build per-skill breakdown. For each discovered verdict, capture:
ref— the eval skill name + mode (e.g.ocs-chatbot-eval-deep)score— the verdict'soverall_scoreverdict— pass / warn / failnote— one-line summary: overall score + weakest-dimension name + its score. Weakest dimension is thedimensionsentry with the lowestscorevalue.
Draft improvement recommendations (LLM-as-Judge). For each verdict that is
warnorfail, and for eachdimensionsentry withscore < 6.0across any verdict, generate a one-line recommendation. Seed the prompt with:- The verdict YAML (full).
- The relevant skill's
## LLM-as-Judge Rubricsection fromskills/<skill-name>/SKILL.mdif present — read withdrive_read_file's local-file equivalent (or by locating the skill definition underskills/). If the skill has no rubric section, emit[INFO] skill <X> lacks a rubric — recommendation will be generic. (That's the forcing function for per-skill rubric work.) - A terse instruction: "Given this verdict, propose one concrete change that would move the weakest dimension toward 8+."
Keep each recommendation to a single sentence, imperative voice, actionable (point at a skill + what to change). Example:
"Tagging dimension consistently below 7.0 across 2 runs — consider tightening [training-gap] application rules in the golden template prompt".
10b. Independent deployability spot-check (fitness backstop — added
2026-05-29). opp-eval's rollup trusts the upstream verdicts, which
means it launders any inflation in them: if every per-skill eval
is conformance-only and inflated, the rollup computes a high number
and stamps PASS (the ITN failure mode — both app evals scored 9.6 on
a hollow build and the cycle grade would have rolled straight up).
To break the self-referential loop at the one funnel point, --deep
and --monitor run one LLM-as-Judge probe against the
highest-stakes raw artifact the opp has produced — decoupled
from every upstream verdict.
- **Pick the artifact** (first available, in priority order): the
built Deliver app Nova blueprint (`get_app` on the deliver
`nova_app_id` from `3-commcare/pdd-to-deliver-app_summary.md`) →
the generated synthetic records → the published solicitation body.
- **The probe prompt** (does NOT see the upstream scores): "Reading
this actual artifact, would a Dimagi domain expert ship it to a
real LLO today? Score 0–10 on deployability and name the concrete
blockers (missing input validation, capture fidelity, case
write-back, language conformance, enforcement — whatever applies)." This
is the same out-of-chain fitness bar the per-skill rubrics now use
(`_eval-template.md § out-of-chain fitness requirement`).
- **Hard-cap rule:** if the deployability probe scores **< 5** while
the rolled-up overall is **≥ 7** (would-be PASS), cap the
run-level verdict at **`warn`** and surface a `[BLOCKER]`:
"rollup says PASS but the independent deployability spot-check
found the artifact undeployable (score X, blockers: …) — the
per-skill evals for this category are inflated; recalibrate before
trusting this cycle grade." Record the probe score + blockers in
the verdict YAML under `deployability_probe`.
- The probe is scoped (one artifact, one judgment) — opp-eval is
still an aggregator, not a full re-grader. This is the single
deliberate exception to "trust the upstream judgments."
Write the machine-readable verdict to
ACE/<opp-name>/runs/<run-id>/10-closeout/opp-eval/opp-eval_verdict-<mode>.yaml(only for--deepand--monitor). Uses the shared verdict shape — see § Verdict YAML Shape below.Write the human-readable scorecard to
ACE/<opp-name>/runs/<run-id>/10-closeout/opp-eval/opp-eval_scorecard-<mode>.md. Shape documented in § Scorecard Output below.
In
--monitormode, append a single-line entry toACE/<opp-name>/runs/<run-id>/10-closeout/opp-eval/opp-eval_trend.mdwith date, overall score, each non-null category score, and the number of verdicts aggregated. One row per run so drift is visible at a glance.Write
phases.closeout.products.opp_evalto the current run'srun_state.yamlso downstream readers (ace-web summary in particular) get the headline score without re-parsing the scorecard markdown.phases: closeout: products: opp_eval: mode: <quick | deep | monitor> overall_score: <0-100 weighted score from Step 7> verdict: <pass | warn | partial | fail | incomplete> scorecard_file_id: <Drive fileId of opp-eval_scorecard-<mode>.md> verdict_file_id: <Drive fileId of opp-eval_verdict-<mode>.yaml — null on --quick> trend_file_id: <Drive fileId of opp-eval_trend.md — null outside --monitor> evaluated_at: <ISO timestamp>Apply via
mcp__plugin_ace_ace-gdrive__update_yaml_filewithmerge: 'deep'(partial phase-child patch —deeppreserves the phase'sstatus/steps;two-levelwould drop them, #572/#587). Sole writer ofproducts.opp_eval. Each re-run replaces theopp_evalkeys in full — this is intended; consumers read the latest scorecard, not the history (the trend file captures history).
LLM-as-Judge Rubric
opp-eval's rubric is aggregation rules, not a per-response grader. The per-skill verdicts already did the LLM-as-Judge grading; opp-eval rolls those scores up and decides what's worth flagging as a recommendation.
Category scoring rules:
- Per-category score = simple mean of the
overall_scorevalues of every verdict whoseskillmaps into that category (see the category table in step 6). - Categories with zero verdicts score
null(not 0). A missing verdict is an information gap, not a zero. - Weights renormalize across the non-null categories so a partial opp isn't penalized for being early in its lifecycle.
Run-level scoring rules:
- Overall = weighted mean of non-null category scores.
- Verdict: pass ≥ 7.0 · warn 4.0–6.9 · fail < 4.0.
Recommendation triggers (one recommendation per trigger):
- Any verdict with
verdict: warnorverdict: fail. - Any
dimensionsentry (inside any verdict) withscore < 6.0. - Any skill-category with only 1 verdict when ≥ 3 are expected at the
opp's current phase — surfaced as
[INFO] thin coveragerecommendation.
What opp-eval will NOT do:
- Re-grade any individual response. That's the per-skill eval's job. One scoped exception (2026-05-29): the deployability spot-check (step 10b) makes a single independent fitness judgment on the highest-stakes raw artifact — not a per-response re-grade, but a backstop against laundering inflated upstream scores into a PASS.
- Invoke any
-qaor-evalskill. Evidence is pre-captured; opp-eval only aggregates (plus the one spot-check above). - Block a gate. The brief exists for uniformity, not gating.
Archetypes
This skill is archetype-agnostic by design. Per-skill -eval
skills already applied archetype-specific rubrics when they graded
their own artifact (e.g., cycle-grade adds the Research Quality
dimension for focus-group PDDs). opp-eval reads the resulting
verdict scores without re-branching on archetype — it trusts the
upstream judgments.
The one place archetype matters is recommendation phrasing: if
the PDD is available, opp-eval reads archetype: in step 1 and
passes it to the LLM prompt in step 10 so recommendations land in the
right domain vocabulary (e.g., "tighten the FGD facilitator
instructions" for focus-group vs. "tighten the visit-flow prompt"
for atomic-visit). If the PDD is missing, recommendations are
archetype-agnostic.
Verdict YAML Shape
Written to
ACE/<opp-name>/runs/<run-id>/10-closeout/opp-eval/opp-eval_verdict-<mode>.yaml
in --deep and --monitor modes. Follows the shared shape from
skills/README.md § QA vs Eval — the two-phase pattern:
skill: opp-eval
target: <opp-name>
mode: deep | monitor
ran_at: <ISO timestamp>
capture_path: runs/<run-id>/ # opp-eval reads a run directory, not a single file
overall_score: 0.0-10.0 # weighted, renormalized across non-null categories
verdict: pass | warn | fail # capped at warn if deployability_probe.score < 5 and raw rollup >= 7
deployability_probe: # independent raw-artifact fitness backstop (step 10b); --deep/--monitor only
artifact: <which raw artifact was judged, e.g. "deliver app blueprint 2VI6...">
score: 0.0-10.0 # out-of-chain "would an expert ship this?" judgment
blockers: ["<concrete blocker>", ...]
capped_verdict: true | false # true if this probe forced the run verdict down to warn
dimensions:
# For opp-eval, "dimensions" are skill categories. A null `score`
# means no verdicts were found in this category; its weight is
# renormalized away from the overall.
design: { score: X.X | null, weight: 0.20 }
commcare: { score: X.X | null, weight: 0.20 }
connect: { score: X.X | null, weight: 0.15 }
ocs: { score: X.X | null, weight: 0.20 }
operate: { score: X.X | null, weight: 0.15 }
closeout: { score: X.X | null, weight: 0.10 }
per_item:
# One entry per verdict file discovered under runs/<run-id>/
- ref: ocs-chatbot-eval-deep
score: 8.4
verdict: pass
note: "Overall 8.4; weakest dim: tagging (6.8)"
- ref: cycle-grade
score: 6.2
verdict: warn
note: "Overall 6.2; weakest dim: research_quality (4.1)"
recommendations:
# One per trigger from the rubric. Severity matches the trigger type.
- for: ocs-chatbot-eval
severity: warn
recommendation: "Tagging dimension consistently below 7.0 across 2 runs — tighten [training-gap] application rules in golden template prompt"
- for: cycle-grade
severity: warn
recommendation: "Research Quality scored 4.1 — consider adding an explicit FGD sampling plan section to the PDD template"
Scorecard Output
Quick-mode (10-closeout/opp-eval/opp-eval_scorecard-quick.md)
# Opportunity Eval — Quick
Opportunity: <opp-name>
Generated: <ISO timestamp>
Current phase (from run_state.yaml): <phase>
## Structural check
- **Present:** N / M required artifacts for phase <phase>
- **Missing:** <list, or "none">
- **Unexpected:** <files in the folder that aren't in the manifest, or "none">
## Notes
<One line per INFO from the structural scan. Concrete examples:
[INFO] run_state.yaml missing — assumed design phase (default)
[INFO] 2 unexpected files are operator-maintained (improvement-backlog.md, iteration-log.md); safe to ignore
[INFO] pdd.md archetype: focus-group — deep-mode recommendations will use FGD vocabulary
If no INFOs to surface, write "None.">
_Run `--deep` to aggregate verdicts and get improvement recommendations._
Stdout summary format (always printed, both quick and deep modes):
opp-eval <mode>: <P>/<M> present, <K> missing, <U> unexpected (phase: <phase>)
Example: opp-eval quick: 4/5 present, 1 missing, 2 unexpected (phase: design)
Deep / monitor (10-closeout/opp-eval/opp-eval_scorecard-<mode>.md)
# Opportunity Eval — <mode>
Opportunity: <opp-name>
Generated: <ISO timestamp>
Overall Score: X.X / 10 · Verdict: <pass | warn | fail>
## Category Breakdown
| Category | Score | Weight | Verdicts counted |
|---|---|---|---|
| design | X.X | 0.20 | N |
| commcare | X.X | 0.20 | N |
| connect | — | 0.15 | 0 (not yet run) |
| ocs | X.X | 0.20 | N |
| operate | X.X | 0.15 | N |
| closeout | X.X | 0.10 | N |
## Per-Skill Results
| Skill | Score | Verdict | Weakest Dimension | Note |
|---|---|---|---|---|
| ocs-chatbot-eval-deep | 8.4 | PASS | tagging (6.8) | Overall 8.4; weakest dim: tagging |
| cycle-grade | 6.2 | WARN | research_quality (4.1) | Overall 6.2; weakest dim: research_quality |
| ... | ... | ... | ... | ... |
## Recommendations
- **[ocs-chatbot-eval · warn]** Tagging dimension consistently below 7.0 across 2 runs — tighten [training-gap] application rules in golden template prompt
- **[cycle-grade · warn]** Research Quality scored 4.1 — consider adding an explicit FGD sampling plan section to the PDD template
## Structural check
- Present: N / M required artifacts for phase <phase>
- Missing: <list, or "none">
## Notes
<Any [INFO] lines: missing PDD, malformed verdict YAMLs, skills without rubrics, thin coverage, etc.>
Gate Brief — reference only, NOT an artifact this skill writes
This skill writes no gate-brief file. The write step was removed in
0.13.116 and lib/artifact-manifest.ts registers no gate-brief artifact.
opp-eval does not gate any phase in any case. What follows specifies the
content an operator (or future automation) should read out of this
skill's scorecard + verdict YAML — it is not a set of write instructions.
Do not emit it as a file.
Source artifacts: 10-closeout/opp-eval/opp-eval_scorecard-<mode>.md +
10-closeout/opp-eval/opp-eval_verdict-deep.yaml.
- Artifact Under Review: path to the scorecard at
ACE/<opp-name>/runs/<run-id>/10-closeout/opp-eval/opp-eval_scorecard-<mode>.md; summary is<overall-score>/10 across <N> verdicts, run-level verdict <pass|warn|fail> - What to Check (emit these 4 items verbatim):
- Overall score ≥ 7.0 and no category-level score below 5.0
- Every expected skill category for the opp's current phase has at least one verdict (no information gaps)
- Each recommendation has a concrete, actionable target — if any recommendation is too vague ("improve quality"), push back
- Skills flagged as lacking a rubric are queued for rubric work before the next opp (forcing function)
- Auto-Surfaced Concerns: one line per signal:
[BLOCKER]if overall verdict isfail[WARN]for each per-skill verdict that iswarnorfail[WARN]for each category whose score is below 5.0[INFO]for each skill without a## LLM-as-Judge Rubricsection[INFO]for each category with no verdicts yet (thin coverage)- "None — all auto-checks passed." if pass with no warns
- Recommended Disposition:
Review(opp-eval never auto-approves or auto-rejects a phase; the brief is advisory by design). Spell outReviewverbatim so downstream readers don't confuse it with the 5 gating skills' Approve/Reject/Iterate set.
MCP Tools Used
- Google Drive:
drive_read_file,drive_list_folder,drive_create_file,drive_update_file - No OCS calls — opp-eval reads pre-captured verdicts, it does not exercise any artifact
Mode Behavior
- Auto: Walk the manifest, aggregate verdicts, write scorecard + verdict. No pause. Print stdout summary.
- Review: Pause after aggregation to show the scorecard before writing the final files. opp-eval is advisory either way — there is no Approve/Reject prompt.
Dry-Run Behavior
opp-eval has no external side effects (no emails, no API calls, no
tickets). In --dry-run mode it behaves identically to a normal run:
reads Drive, writes scorecard + verdict + brief to Drive. These are
all human-facing artifacts; Drive writes are not treated as effectful
for dry-run purposes (same convention as ocs-chatbot-eval).
Change Log
| Date | Change | Author |
|---|---|---|
| 2026-04-19 | Initial version — umbrella eval aggregator. Three modes (--quick / --deep / --monitor); rolls every verdicts/*.yaml into a run-level scorecard across 6 skill-category dimensions, emits improvement recommendations, and writes a uniform-contract gate brief (advisory; does not gate a phase). |
ACE team (opp-eval rollout) |
| 2026-04-19 | Quick-mode template: add Unexpected: row (skill was already surfacing unexpected files but the template omitted it); tighten Notes wording with three concrete examples; specify stdout summary format including unexpected count. Surfaced in first run against a real partial opp (cosmetics-fgd-pilot) |
ACE team (qa/eval iteration loop) |
| 2026-05-05 | Path-scheme migration on the umbrella aggregator. Discovery (Step 5) walks each phase folder under runs/<run-id>/ collecting *_verdict*.yaml (instead of ACE/<opp>/verdicts/*.yaml, which is gone). Outputs land under 10-closeout/opp-eval/: scorecard (opp-eval_scorecard-<mode>.md), verdict (opp-eval_verdict-<mode>.yaml), gate brief (opp-eval_gate-brief-deep.md), and --monitor trend (opp-eval_trend.md). Verdict YAML capture_path field updated to runs/<run-id>/. Wiring fix — prior glob would have found zero verdicts on a fresh run. No behavior change beyond paths. |
ACE team |
| 2026-05-29 | Anti-laundering deployability backstop (ITN post-mortem). Added step 10b: an independent LLM-as-Judge deployability spot-check on the highest-stakes raw artifact (deliver blueprint → synthetic records → solicitation), decoupled from all upstream verdicts. If it scores < 5 while the rollup is ≥ 7, the run verdict is capped at warn with a [BLOCKER]. Previously opp-eval was a pure score-rollup with no independent fitness term, so inflated per-skill scores (the ITN 9.6s) laundered straight into a cycle PASS. New deployability_probe block in the verdict YAML. The one scoped exception to "trust the upstream judgments." Per docs/superpowers/specs/2026-05-29-eval-fitness-gap.md. |
ACE team |
Claims
If the run folder holds claims.yaml, roll its tally into the scorecard via summarizeClaims
(lib/run-claims.ts) and render the section with renderClaimsSection
(lib/render-claims.ts).
The scorecard is an INTERNAL artifact, so call it as
renderClaimsSection(set, { audience: 'internal' }) — that is the only audience that
carries each claim's audit evidence. The DEFAULT is 'counterpart', which renders the
says sentence and never evidence: Drive file ids, MCP atom signatures and scraped-field
reliability caveats are the class skills/agent-turn-review § F bans from anything a
counterpart reads (ace#2386, ace#2420). What the counterpart herself sees is the run's
ace-web page, which renders the same claim set to the same contract.
all_met is TRUE only when EVERY claim is MET. A run where every ANSWERED claim passed but a
checkpoint never ran has not met its claims — NOT REACHED accuses, it never passes, and
reporting otherwise recreates the silence the mechanism exists to remove. Design:
docs/superpowers/specs/2026-09-15-pre-run-claims-post-run-validation-design.md.