reporting-results
Writes what an AQVEN project learned for readers outside the chat: the EXPERIMENTS.md journal, holdout findings apart from dev signals, short reports. Use first when asked what was learned, for a summ
SKILL.mdpackages/aqven/src/aqven/agent_plugin/skills/reporting-results/SKILL.md
GitHubMUST
- A report opens with the result on real data: the number, the change in percentage points against the baseline, and n. Synthetic tests only support it, after it.
- Reports and explanations of a concept open with two or three sentences; details only when asked. An internal variant or check id only with a plain explanation next to it.
- Holdout findings (
FINDINGS.md) stay apart fromdevsignals; adevsignal is never called a finding. FINDINGS.mdis generated by the engine: never edit it.<package>/EXPERIMENTS.mdis yours to write.- Quote
verdict.textas it is; never round or retell its numbers. - Before you report a comparison, open lost and won cases with their inputs (step 4).
- Spend in the journal is measured:
spend.usdofseries_getper series,series_liststatsfor totals; an estimate appears only before a run and is labelled as one. - Summaries and statistics for the owner are a Markdown file in
<package>/, next toEXPERIMENTS.md, unless the owner asks for another format. - An experiment whose question is answered gets
archived: truein itsexperiment.yaml, the only edit allowed on an experiment that has series; it is never deleted.
Procedure
| # | Step | Exit criterion |
|---|---|---|
| 1 | After every series that ends done: an entry in <package>/EXPERIMENTS.md with the date, experiment and series id, the question, verdict.text quoted, the numbers with n, Scope (what the claim covers and what it does not), Read (the cases you opened and what you saw), Spend (measured) and Changes |
the entry exists with every part |
| 2 | Sections of the journal: Now (the owner's goal and the stage), Planned (the owner's hypotheses and data policies not run yet), What holdout settled (pointing at FINDINGS.md), What works (dev signals), What does not, How we measure (checks, ground truth, controls), Open questions, Spend (per series, and totals from series_list stats: spend with judge checks, requests, tokens, wall time) |
a reader without the conversation knows what is known and what is planned |
| 3 | A refuted belief is marked with the date, the series and its scope, never deleted. The failure of one implementation is "implementation-level" until a strong version also fails | the history of decisions stays visible; no general lesson rests on one weak implementation |
| 4 | Before reporting a comparison, open 5 to 10 cases the new variant lost and won against the current best, side by side with their inputs: series_outputs {series_id, split: "dev", variant: <candidate>} and the same for the current best, with fields for the decision field and the node the claim is about; run_get_node for a prompt. Held-out outputs are not read per case (series_outputs and series export leave them out), so a holdout entry cites the dev reading. A breakdown by tag or label shows n in every cell; a cell under 10 cases shows a direction, not a result |
the Read line says in one sentence what you saw |
| 5 | Report to the owner: two or three sentences with the real-data result, the delta in points and n; then the next step and the measured spend | the report passes the MUST rules |
| 6 | Autonomy limit: count experiments since the owner's permission; at the limit, report, update the journal and stop | the owner is called in time |
| 7 | After a compaction, read the journal and invoke the skill of the current stage before any action | facts of the summary checked against the journal |
| 8 | An experiment whose question is answered (a holdout verdict, or the owner closed the question): archived: true in its experiment.yaml, and the journal entry says so |
Studio's Research lists only open questions; the folder and its findings stay |
Entry format:
### 2026-09-24 · intent_escalation_agents · series 0199f1e2-… · holdout
Question: does Qwen as the escalation agent get the intent right at most 0.1 less often than DeepSeek, with no more invalid first outputs and at most 25% slower at p95?
Verdict: "<verdict.text as the series wrote it>"
Numbers: intent right 0.86 against 0.89 (-3 points, within the 10-point margin), valid first outputs 0.97 against 0.95, p95 12% faster, n = 40 support cases.
Scope: the `escalate` node only, with the recorded triage as its input; another agent or another triage needs its own series.
Read: on dev series 0199e5c4, 7 lost and 4 won cases side by side; the losses are messages whose defect claim the rest of the message takes back.
Spend: $0.42 measured.
Changes: the `escalate` node points at qwen; the experiment is archived.
Pitfalls
| What goes wrong | Do instead |
|---|---|
| Asked for progress in a few sentences with percentages, the first answer opened with a synthetic sweep and mixed ranges from different tests | open with real data: the number, the delta in points, n |
| A two-line question about a concept got a four-section answer | two or three sentences, then offer details |
| The owner had to ask for one written conclusion that a person or a model can read outside the conversation | keep EXPERIMENTS.md current after every series |
| The owner had to ask whether anyone had read the model answers | open lost and won cases before every report (step 4) |
| One intermediate record failed (a call summary without speaker turns), and "intermediate steps do not help" went into the journal | a Scope line; one implementation's failure is "implementation-level" until a strong version also fails |
| A per-tag breakdown put 100% on 3 invoices of one layout next to 72% on 60 others, and it was read as a winner | n in every cell; under 10 cases is a direction, confirmed by an experiment on that tag |
| The journal carried pre-run estimates as spend | measured spend.usd from series_get, totals from series_list stats |
| A statistics summary was built as an HTML page | a Markdown file next to EXPERIMENTS.md |
| The journal was rewritten by a script, and refuted entries and hand-written context were lost | Edit or Write, one change at a time |
| A compaction summary carried a false rule for hours | the journal is the memory; re-read it after a compaction |
| Answered experiments left among the open ones, so the owner could not tell what was still in question | archived: true once the question is answered |
A good pattern: allowed a set number of experiments with a report at the end, the agent stayed within it and wrote a journal that opens with the main lesson and keeps the one finding apart from dev signals |
do the same |
Tools and commands
- Read, Edit and Write for
<package>/EXPERIMENTS.mdand owner summaries; never a script or a heredoc. Edit forarchived: trueinexperiments/<id>/experiment.yaml. aqvenMCPseries_getfor the verdict, the matrix andspend.usd(view: "summary"first); withinclude_cases: truefor the case rows of adevseries.series_outputs(split: "dev",variant,case,fields) for what each variant answered.series_list(experiment_id) for the history andstats.run_get_nodefor the prompt and the answer of one attempt.
References
references/concepts/research-journal.md: the journal's sections, a sample entry, the rule for owner reports. Read before the first entry in a project.references/reference/findings.md: what a finding file andFINDINGS.mdhold. Read before pointing at a finding.references/concepts/experiments-series-and-findings.md:devsignals against holdout findings. Read when unsure whether a result is a finding.references/mcp-cli/experiments-and-series.md:series_outputsfilters andfields,series_liststats. Read at steps 2 and 4.