demo-fidelity-check
Pass/fail check that a cloned synthetic dataset reproduces the real source's statistical shape before a funder sees it. Needs the clone and its source opp.
SKILL.mdskills/demo-fidelity-check/SKILL.md
GitHubDemo Fidelity Check
Runs only in the clone provider path of demo-data-setup. A cloned demo
claims to mirror a real program — so before it's shown to a funder, verify the
synthetic fixtures actually reproduce the source's distributions. A low-fidelity
clone shown as "this is the real program" is the worst failure mode this pipeline
has; this gate prevents it.
See skills/_qa-template.md for the shared QA contract (verdict YAML, auto-fix
protocol, static-vs-LLM rules).
Inputs
| Parameter | Used for | Resolver |
|---|---|---|
bundle_dir |
the profile bundle under fidelity check (from synthetic_profile_from_prod / synthetic_clone_profile) |
explicit arg; default: the demo-data-setup (clone) handoff |
provider |
must be clone; anything else skips |
explicit arg; default: run_state …products.synthetic.source |
Given bundle_dir, this runs on any cloned dataset, with or without a demo run.
Products
<demo-run>/7-synthetic/demo-fidelity-check_result.yaml— QA result perlib/qa-types.ts
Process
- Provider guard. If
source.provider != clone, emitverdict: skippedwith detail "fidelity check is clone-only" and stop. (denovo/ace-run invent their own shape; there is no real source to compare against.) - Run the report.
mcp__connect-labs__synthetic_fidelity_report(bundle_dir)— returns per-field mean/std/TVD deltas and a correlation Frobenius distance between the generated fixtures and the manifest they came from. - Score against thresholds (below). Any BLOCKER breach →
fail; elsepass.
Checks
| # | id | type | description | auto-fix on fail |
|---|---|---|---|---|
| 1 | report_ran |
static | synthetic_fidelity_report returned a report for the bundle |
re-check bundle_dir path; re-profile if the bundle is missing |
| 2 | field_distributions_close |
static | every profiled field's TVD ≤ 0.15 and |mean delta| within tolerance (tune per rubric) | regenerate with mirror=true (or pin form_json_paths) so the clone tracks the source shape |
| 3 | correlations_preserved |
static | correlation Frobenius distance ≤ threshold — the clone preserves cross-field structure, not just marginals | use mirror=true (transplant pool) — marginal-only mode loses per-entity structure |
| 4 | trajectories_present |
static | for a longitudinal demo (e.g. per-child MUAC recovery), each entity has its multi-visit series (mirror pool populated) | re-profile with mirror=true; marginal mode collapses trajectories |
Thresholds are provisional pending calibration against ≥3 real clones (mark the
rubric provisional per docs/eval-calibration-learnings.md). On fail, the safe
fallback is to author the demo denovo rather than show a low-fidelity clone.