Validation Reality — what counts as evidence in nuxt-crouton

One-line purpose: the verified test-running commands, the honest coverage map, and the gap between claimed and actual gating — so you know what a green check does and does not prove.

When to use / when NOT to use

You want	Go to
Run/interpret the fixture smoke end-to-end, pick a fixture, triage its failures	e2e-smoke skill (but see the staleness note in §2 — several of its facts have drifted)
The test-first sign-off gate — proposing a failing test for human approval before writing packages/* logic	test-review skill
Fixture manifest schema, auth-helper internals, adding a fixture	e2e/CLAUDE.md + fixtures/CLAUDE.md (canonical)
Which CI workflow a failing check belongs to, deploy pipeline	crouton-ci-and-deploy-map skill
A test fails with an unfamiliar error string	crouton-diagnostics-index skill
Fresh clone won't build/boot before tests can run	crouton-build-and-env skill
This skill	Run any test kind; know what coverage actually exists; know what CI enforces

1. What counts as evidence here

The house rule: done is signed off, not asserted (AGENTS.md) — green proxies each lied during a real graduation (#988; full story: sibling crouton-failure-archaeology).

Concrete local example: a staging preview deployed green but with an empty database, so a loading-skeleton change "showed no difference" — the deploy proved nothing until the advertised path was walked with real data (#695). So the evidence ladder, weakest → strongest:

1. pnpm typecheck green — proves types compile, nothing about behaviour.
2. pnpm test green — proves the existing assertions pass (totals: the run summary); see §4 for what has zero assertions.
3. E2E fixture smoke green — proves a generated app boots → authenticates → does CRUD → mounts package surfaces; it does not deeply drive package UIs (e.g. with-pages mounts the editor workspace but never types into it).
4. Walking the advertised path yourself (the verify skill's job) and a human sign-off — the only "done".

2. Run every test kind (commands verified — see Provenance)

Kind	Command	Notes
Unit, all	pnpm test	Vitest. Root vitest.config.ts is a projects config: projects: ['packages/*/vitest.config.ts'] — only packages shipping their own vitest.config.ts participate (count them: ls packages/*/vitest.config.ts). Suite totals (passed/skipped/todo) come from the run summary — re-verify block.
Unit, one package	pnpm --filter @fyit/crouton-i18n test	Each participating package has "test": "vitest run".
Unit, watch / coverage / UI	pnpm test:watch · pnpm test:coverage · pnpm test:ui	Coverage is v8, per package config.
E2E fixture smoke	E2E_FIXTURE=<name> pnpm test:e2e	= playwright test --config e2e/playwright.config.ts; default fixture minimal; one fixture per run (all bind :3000). Full workflow → e2e-smoke skill.
E2E, subset of specs	pnpm test:e2e e2e/collection.smoke.spec.ts e2e/surface.smoke.spec.ts	The setup project (login) always runs first via project dependency.
Typecheck (the gate)	pnpm typecheck	= pnpm -r --filter './apps/*' typecheck — apps only. NEVER npx nuxt typecheck from root (no app context → thousands of false positives; root CLAUDE.md; mechanics in crouton-build-and-env §4).
Typecheck fixtures	pnpm typecheck:fixtures	The #197 generator-regression gate surface.
Typecheck MCP	pnpm typecheck:mcp	= pnpm --filter @fyit/crouton-mcp typecheck (was a silent no-op with a stale filter name until fixed under #1098). Stale-name inventory: crouton-config-registry § "Silent no-ops".
MCP server tests	pnpm --filter @fyit/crouton-mcp build && pnpm --filter @fyit/crouton-mcp test	Build first — CI does (ci.yml mcp-server-tests job).
Lint	pnpm lint / pnpm lint:fix	eslint . from root, eslint.config.mjs. No CI workflow runs repo-wide pnpm lint (the misnamed "Lint & Type Check" job was renamed typecheck-mcp under #1097; CI still runs no eslint).
Publish validation	pnpm check:publint · pnpm check:attw	Both target ./packages/crouton only.

E2E environment realities (verified in e2e/playwright.config.ts):

• Auth env is self-provided: webServer.env defaults BETTER_AUTH_SECRET=dev, BETTER_AUTH_URL=http://localhost:3000; fixtures also commit a dummy .env. Exported vars override (how CI injects its own). Exporting them is harmless.
• The REAL timeouts are deliberately huge — per-test/navigation/webServer-boot in the minutes, expect/action in the tens of seconds (read current values with the re-verify grep). The smoke runs against nuxt dev, and cold route compiles on CI take minutes; tightening them is what made the smoke flake (comment in the config itself).
• Sandboxes with blocked browser downloads: PW_EXECUTABLE_PATH=/opt/pw-browsers/chromium-<build>/chrome-linux/chrome is a committed escape hatch in the config.
• Fresh worktree prerequisite: pnpm --filter "e2e-fixture-<name>^..." build (dist-consumed @fyit/* deps must exist).
• Shell trap (reproduced): pnpm test \| tail masks the real exit code — check output content, not $?.

Staleness in the e2e-smoke skill

Its workflow is correct, but several of its facts have drifted — stale timeout claim, fixture table missing the newer fixtures, over-strict "env vars are required" claim — tracked as #1103 (row 7). e2e/CLAUDE.md carries the current story; trust it over the skill's numbers.

3. The fixture harness — what exists and what each proves

Vocabulary: a fixture is a real generated crouton app under fixtures/<name>/ (throwaway, never deployed) that the Playwright harness in e2e/ boots and smokes. What each fixture smokes is declared in its fixtures/<name>/e2e.manifest.json; the specs are generic and manifest-driven — you almost never write per-fixture test code. Full manifest schema + auth internals: e2e/CLAUDE.md (canonical). Delta this skill carries:

Fixtures on disk vs in CI's ALL list — count both fresh (ls fixtures/; grep 'ALL=' .github/workflows/e2e.yml). Snapshot 2026-07-02 — regenerate before relying on it:

Fixture	Adds	In CI ALL?
minimal	core + auth + i18n, one mainItems collection	✅
with-pages	crouton-pages → transitively crouton-editor	✅
with-bookings	crouton-bookings	✅
with-assets	crouton-assets	✅
with-maps	crouton-maps	✅
with-devtools	crouton-devtools + crouton-feedback	✅
with-collab	crouton-collab	❌ deliberately excluded — pre-existing type errors fail the #197 typecheck gate; #210 keeps it a local-only spike (comment in e2e.yml)

Manifest keys (typed as FixtureManifest in e2e/helpers.ts): packages (drives smart CI fixture selection, #622), collections[] (key, heading, create, optional update/requiredField/a11y), optional surfaces[], i18n, maps, root a11y. Real example — fixtures/minimal/e2e.manifest.json declares one collection (mainItems, requiredField: "name") plus an i18n flip ("Your teams" → "Jouw teams").

Auth realities in specs (the non-obvious ones; verified in e2e/helpers.ts / e2e/auth.setup.ts):

Reality	Consequence
Login/register is a RouteModal overlay, not a form page	Helpers fill inputs inside the modal; /auth/login redirects to / and opens it
Signup creates no team	ensureTeam() POSTs better-auth org endpoints, which need an Origin header (CSRF)
Better-auth 415s bodyless POSTs	Always send Content-Type: application/json (signOut() passes data: {})
get-session is cookie-cached (5-min TTL)	Force ?disableCookieCache=true for true state; setup strips the session_data cookie from saved storageState so slow runs don't SSR logged-out
Sign-out kills the shared session	Auth specs mint their own browser contexts; content specs call ensureAuthed() (self-healing re-login)
Test identity	e2e-user@example.com / TestPassword123!, team e2e-team (constants in e2e/helpers.ts)

A11y in the smoke (verified in e2e/helpers.ts): axe runs on every collection list + surface; only critical/serious block; color-contrast excluded (theme-owned); baseline of known upstream shell violations = ['aria-allowed-attr'] (#735) — button-name was driven to zero and removed, so regressions there now fail.

CI e2e (.github/workflows/e2e.yml): smart per-PR fixture selection by changed paths (#622); packages in UNIVERSAL (crouton crouton-core crouton-i18n crouton-cli crouton-auth) → full matrix; push to main + nightly cron → always full matrix. Runs in the mcr.microsoft.com/playwright container whose tag must track the repo's @playwright/test version (read both in e2e.yml + the lockfile). Each job regenerates the fixture from committed schemas (crouton-generate.js config --force) and typechecks the regenerated output (the #197 generator-regression gate) before testing — so after changing a generator template, regenerate fixtures and commit, or CI diverges from your tree.

4. The honest coverage table

Test files per package — *.test.ts/*.spec.ts excluding node_modules/dist/.nuxt/.output. Snapshot 2026-07-02 — regenerate with the re-verify block before relying on numbers. At the snapshot, fewer than half the packages had any tests (14 of 31; 17 at zero):

Package	Test files		Package	Test files
crouton-core	29		crouton-sales	2
crouton-layout	17		crouton-mcp	2
crouton-cli	15		crouton-analytics	2
crouton-auth	14		crouton-assets	1
crouton-flow	8		crouton-feedback	8
crouton-collab	6		crouton-printing	5
crouton-i18n	4		crouton-triage	3

Zero unit tests at the snapshot: crouton (meta), crouton-admin, crouton-ai, crouton-audio, crouton-bookings, crouton-charts, crouton-devtools, crouton-editor, crouton-email, crouton-events, crouton-maps, crouton-mcp-toolkit, crouton-pages, crouton-themes, crouton-three.

Mitigations and their limits:

• pages/bookings/assets/maps/devtools get e2e fixture smoke coverage (§3); editor only transitively (with-pages mounts the workspace). collab has unit tests but no CI e2e (excluded fixture) — and its one component test file is describe.skip'd with a note that it "tests an inline mock component, not the actual CollabEditingBadge.vue" (packages/crouton-collab/tests/components/CollabEditingBadge.test.ts).
• apps/* and pocs/* have 0 test files (verified by find). Consistent with the stage model (node scripts/harness-stages.mjs apps/velo → test-first opt-in; pocs/* → off) — but it means launched apps (fanfare/velo/triage) have no automated behaviour tests beyond typecheck and fanfare's dual-preset CI build.
• The tests that exist are real, not placeholders (spot-checked: crouton-core/tests/api/*.test.ts, crouton-layout/app/utils/__tests__/layout-viability.test.ts).

Gotcha (reproduced at the snapshot): packages/crouton-charts ships a vitest.config.ts but zero test files — standalone npx vitest run there prints "No test files found, exiting with code 1", while the root projects run tolerates the empty project (root pnpm test is green). Anyone adding --project filtering will hit this.

5. The skipped-suite inventory — three mocking walls

The run summary reports a large skipped + todo tail (exact totals: pnpm test 2>&1 | tail -4; marker counts: the greps in the re-verify block). The dormant cases cluster behind three mocking walls — fixing these mocks is the single highest-leverage test-debt target (on the order of 170 dormant cases at the snapshot, many asserting security-relevant behaviour like team limits):

Wall	What's dormant	Files (verified describe.skip / todo locations)
Better Auth nanostore (+ window.location, WebAuthn)	crouton-auth's entire integration suite (login + registration flows) whole-file skipped; a stack of it.todos in useTeam/useAuth	packages/crouton-auth/tests/integration/auth-registration.test.ts, auth-login.test.ts, tests/unit/composables/useTeam.test.ts, useAuth.test.ts
Nuxt #imports / useRuntimeConfig	team-utils cases incl. team-limit / allowCreate / single-tenant enforcement; useScopedAccess (import.meta.client unmockable post-import)	packages/crouton-auth/tests/unit/server/team-utils.test.ts, tests/unit/composables/useScopedAccess.test.ts
Yjs / WebSocket	crouton-flow's entire realtime-sync feature — unit + integration whole-file skipped, e2e suite skipped, dagre-mock describe.todos → the sync feature has zero passing coverage of any kind	packages/crouton-flow/test/composables/useFlowSync.test.ts, test/integration/flow-sync.test.ts, test/e2e/multiplayer-flow.spec.ts, test/composables/useFlowLayout.test.ts

Smaller items: crouton-core useCollectionQuery/useCollectionMutation logging todos/skips; crouton-i18n/app/composables/__tests__/zod-sanity.test.ts encodes a do-not-use Zod v4 pattern (z.record(valueSchema) breaks — use the two-arg form) as a skipped sanity test.

6. What CI actually gates vs what CLAUDE.md claims

ci.yml jobs:

Job	What it actually does	Gap vs the claim
typecheck-mcp + typecheck-apps	MCP-server typecheck, and the full app sweep (pnpm typecheck)	Fixed under #1097: typecheck-apps builds the apps' workspace deps, nuxt prepares each app, and runs the exact local sweep — root CLAUDE.md's "EVERY change requires pnpm typecheck" is now a CI gate too (was: lint-and-typecheck, MCP-only, no eslint — renamed since it overclaimed). e2e.yml's regenerated-fixture typecheck (#197) still covers fixtures; the typecheck:mcp script's stale filter was fixed under #1098.
build-fanfare	Builds fanfare for cloudflare-pages + node-server presets	Fanfare typecheck intentionally NOT gated ("known pre-existing baseline of type errors" — comment in the job); the build is the smoke.
test	pnpm install --ignore-scripts → builds crouton-auth/crouton-core/crouton → npx nuxt prepare in each apps/* → pnpm test	This is the real unit-test gate. Other suited packages (layout, cli…) run from source; the three builds are for dist-consumers.
mcp-server-tests	Build then test @fyit/crouton-mcp	—
changeset-check	Detects packages/ change without a changeset	Emits only ::warning — never fails a PR.
docs-check / sync-validation / package-check	Docs audit script / field-types sync / publint	—

Also: repo-wide pnpm lint runs in no workflow (part of the #1097 picture). E2E gating lives in e2e.yml (§3), not ci.yml. For the full workflow map, defer to crouton-ci-and-deploy-map.

Trust order when these conflict: see crouton-docs-trust-map §1. CLAUDE.md's typecheck-everything rule stands as the required working practice; just don't claim CI enforces it.

7. Adding a test the house way

1. Check the gate first: node scripts/harness-stages.mjs <path> → packages/* = test-first required; apps/* opt-in; pocs/* off. For new packages/* logic, the order is owned by the test-review skill: enumerate edge cases in plain language → write the failing test → hold for a lgtm/approve comment (not a reaction/label, #572) → implement to green.
2. Placement: match the neighbours — *.test.ts beside the source or in __tests__/; packages/crouton-core is the reference layout (this is what test-review itself points at).
3. If the package has no vitest.config.ts, add one (copy a sibling, e.g. crouton-i18n) — the root projects glob packages/*/vitest.config.ts picks it up automatically; nothing to register at root.
4. Run: pnpm --filter @fyit/<pkg> test, then root pnpm test, then pnpm typecheck.
5. E2E coverage for a package surface: extend the relevant fixture's e2e.manifest.json (generic specs pick it up) rather than writing a new spec; new fixture → e2e/CLAUDE.md "Adding a fixture" — and remember CI never runs it until its name is appended to ALL in e2e.yml.

Provenance and maintenance

verified: 2026-07-02

pnpm test 2>&1 | tail -4                                   # suite totals / skipped / todo
ls packages/*/vitest.config.ts | wc -l                     # participating packages
grep -n 'ALL=' .github/workflows/e2e.yml                   # CI fixture list (has with-collab returned?)
grep -nE 'timeout|Timeout' e2e/playwright.config.ts        # real e2e timeouts
for p in packages/*/; do echo "$(find $p -path '*node_modules*' -prune -o -type f \( -name '*.test.ts' -o -name '*.spec.ts' \) -print | wc -l) $p"; done | sort -rn | head -20   # coverage table
grep -rn 'describe\.skip' packages/*/test*/ packages/*/tests/ 2>/dev/null | grep -v node_modules      # mocking-wall status
grep -rnE '\b(it|describe|test)\.(todo|skip)\(' packages/*/test*/ packages/*/tests/ 2>/dev/null | grep -v node_modules | wc -l  # dormant-marker count
pnpm typecheck:mcp                                         # should run tsc in crouton-mcp (fixed #1098); "No projects matched" = regressed