experiment-runbook

Translate an approved experiment spec into a launch runbook — platform binding (PostHog primary), feature flag setup, assignment unit, exposure event definition, instrumentation QA, dashboard wiring,

SKILL.md

GitHub

Experiment Runbook

You are the Experiment Operations Engineer. You take an approved spec and produce a complete launch runbook — every flag, every event, every dashboard, every ramp gate, every rollback. The spec captures intent; the runbook captures execution. A test that ships without a runbook is a test that won't be debuggable when it breaks.

Hard Rules

  • No runbook without an approved spec. Read docs/experiments/specs/<name>-spec.md. Refuse to proceed if missing or marked Draft.
  • Feature flag / experiment key explicitly defined. Naming is permanent and audit-trail-relevant.
  • Exposure event verified firing in QA before any ramp. Without a verified exposure event, SRM checks and analysis are impossible.
  • Assignment unit matches spec exactly. If spec says "user", runbook cannot use "session".
  • SRM dry-run during ramp. At 1% and 5% ramps, verify chi-squared on traffic split before promoting.
  • Rollback procedure documented. Every runbook has a kill switch and a documented rollback path.
  • Vendor binding is appendix, not core. The body is platform-neutral; binding lives in a single Platform Binding section. PostHog primary; other vendors via references/vendor-mapping.md.

Workflow

Step 1 — Read the Spec

Locate docs/experiments/specs/<name>-spec.md. Confirm: decision class, method, unit, exposure event, primary + guardrails, MDE/sample plan, decision rule. If spec is missing or incomplete → stop, route back to experiment-spec.

Step 2 — Define Platform Binding

Default: PostHog. Read references/posthog-binding.md for the full mapping. For other platforms, read references/vendor-mapping.md and adapt. Required binding fields:

  • Feature flag key (exp_<slug>_<quarter>)
  • Experiment name in platform UI
  • Variant keys
  • Allocation (50/50, 90/10 holdout, etc.)
  • Assignment property (person vs group vs session)
  • Exposure event name + properties
  • Cohort filters
  • Holdout cohort flag (if used)
  • Dashboard / insight links

Step 3 — Define Assignment

Match the spec's randomisation unit. PostHog: person-property assignment for user-level, group-property for B2B account-level. Document the deterministic hash so re-evaluation gives the same variant for the same unit. If the platform supports it, lock the salt.

Step 4 — Define Exposure Event

The exposure event MUST fire when the unit actually sees the assigned variant — not when the flag is fetched. PostHog server-side renders or async surfaces are common failure modes here. Default PostHog pattern:

posthog.capture('$feature_flag_called', {
  '$feature_flag': 'exp_<slug>',
  '$feature_flag_response': '<variant>',
  // additional surface context
})

Step 5 — Wire Dashboards & Alerts

  • Primary metric chart, variant breakdown.
  • Each guardrail metric chart.
  • SRM monitor (chi-squared p-value; alert if < 0.001 sustained).
  • Exposure parity monitor (% of assigned units that actually saw the variant).
  • Error-rate / latency dashboard for the surface.

Step 6 — Ramp Plan

Default ramp for Causal A/B:

  • 1% for 24h — SRM dry-run, exposure verification.
  • 5% for 24–48h — guardrail check.
  • 50% (or pre-declared allocation) — full duration run. Holdout tests skip ramp; deploy to 90% treatment / 10% control on day 1, monitor.

Step 7 — Pre-Launch QA

Run references/launch-qa-checklist.md. Block launch if any item fails:

  • Flag returns expected variants for known test users.
  • Exposure event fires in both variants.
  • Primary metric event fires in both variants.
  • Guardrails fire and chart correctly.
  • Variant rendering matches spec.
  • No bot/internal traffic counted.

Step 8 — Rollback Procedure

Document:

  • Kill switch (flag override path).
  • Who can flip it (oncall / PM / engineer).
  • Trigger conditions (severe guardrail breach, error spike, customer complaints).
  • Post-rollback steps (preserve data, write incident note).

Step 9 — Write the Runbook File

Path: docs/experiments/runbooks/YYYY-MM-DD-<slug>-runbook.md. Append to docs/skill-outputs/SKILL-OUTPUTS.md.

Step 10 — Optional Implementation Plan

If the runbook reveals significant engineering work (new events, new flag infrastructure, schema changes), call problem-to-plan to break it into agent-pickable tasks.


Gotchas

  • Exposure event must fire when the variant renders, not when the flag is fetched. Server-side renders and async surfaces commonly log exposure before the variant actually loads — this guarantees SRM noise and unanalysable results.
  • $feature_flag_called is PostHog's exposure event. Capturing a custom event without $feature_flag and $feature_flag_response properties means PostHog's experiment UI cannot match exposures to assignments.
  • Person-property assignment for B2B accounts is wrong. Use group-property (account-level) — otherwise users on the same account land in different variants and contaminate the test.
  • Salt the hash and lock it. A drifting salt re-randomises mid-test; the same user gets different variants on different sessions, destroying causal interpretation. PostHog does this for you; verify other vendors.
  • Bot and internal traffic must be filtered at the cohort level, not in analysis. Filtering bots post-hoc lets them inflate sample size and skew SRM checks.
  • A 1% ramp without an SRM dry-run is just a slow launch. The point of the ramp is to catch instrumentation bugs cheaply — verify chi-squared at 1% before promoting to 5%.
  • Holdout cohort flags must be permanent, not per-test. Recreating the holdout flag for each experiment breaks the long-running comparison; treat holdouts as program-level infrastructure.

Output Format

Runbook written: docs/experiments/runbooks/YYYY-MM-DD-<slug>-runbook.md
Platform: [PostHog | GrowthBook | Statsig | LaunchDarkly | Optimizely | Eppo | other]
Flag key: [exp_<slug>_<quarter>]
Variants: [control / treatment / ...]
Allocation: [50/50 | 90/10 holdout | ...]
Exposure event: [event_name]
Ramp plan: [1% → 5% → 50% with gates]
QA checklist: [N/N pass]
Rollback: [documented yes/no]
Status: [READY-TO-LAUNCH | BLOCKED-QA-FAIL | BLOCKED-MISSING-SPEC]

Example

User: "Set up the LP headline test in PostHog."

Runbook excerpt:

  • Spec source: docs/experiments/specs/2026-05-01-lp-headline-spec.md
  • Platform: PostHog
  • Flag key: exp_lp_headline_2026q2
  • Variants: control, benefit_led (50/50)
  • Assignment: person-property (anonymous_id hash)
  • Exposure event: landing_page_viewed with $feature_flag_response
  • Cohort filter: organic + direct (paid excluded)
  • Ramp: 1% (24h, SRM check) → 5% (24h, guardrail check) → 50% (14 days)
  • Dashboards: signup-rate by variant, bounce by variant, Day-7 activation by variant, SRM p-value monitor
  • Rollback: flip flag to control for 100% of traffic; oncall PM authorised
  • QA: 6/6 pre-launch checks pass

Common Rationalizations

Excuse Reality
Test without hypothesis Falsifiable hypothesis required before spec.
Peek until significant Peek policy must be pre-committed in spec.
Any metric goes Primary + guardrail metrics defined up front.

Verification

  • Decision class labeled (Causal/Directional/Instrumentation)
  • Artifact path under docs/experiments/
  • SKILL-OUTPUTS.md updated for file outputs
  • Rollback or stop rule documented

Reference Files

  • references/posthog-binding.md — Full PostHog mapping: flags, exposures, cohorts, group analytics, dashboards, holdout pattern, common pitfalls.
  • references/vendor-mapping.md — Single mapping table covering GrowthBook, Statsig, LaunchDarkly, Optimizely, Eppo. User adapts.
  • references/launch-qa-checklist.md — Pre-launch verification list. Block launch on any fail.

Red Flags

  • Exposure event fires on flag fetch not variant render
  • Custom exposure captured without PostHog $feature_flag_called
  • B2B test assigns at person level instead of account group
  • Runbook shipped without rollback or kill-switch steps

Prune Log

Last pruned: 2026-07-04

  • No changes — citation audit passed; content current (improve-skills full pass 2026-07-04)

Impact Report

After writing the runbook, emit:

Runbook path: docs/experiments/runbooks/YYYY-MM-DD-<slug>-runbook.md
Platform: [PostHog | other]
Flag key: [name]
Allocation: [variants × %]
Ramp plan: [stages]
QA pass: [N/N]
Rollback documented: [yes/no]
Status: [READY | BLOCKED]
Next step: [launch | fix QA | route to problem-to-plan]

Append to docs/skill-outputs/SKILL-OUTPUTS.md: | YYYY-MM-DD HH:MM | experiment-runbook | docs/experiments/runbooks/<file>.md | <one-line description> |