pod-e2e — test a worktree against an isolated full-stack pod

A worktree's full stack (backend API and frontend SPA) runs as one process on one port — exactly like a Docker container. The kirocrew pod CLI is the only interface you need: spin one up, get a {base_url, token} handle, test against it, tear it down and have the teardown VERIFIED. The live gateway is never touched.

Quickstart — run the bundled e2e suite

bash <app-skills-dir>/pod-e2e/scripts/pod-e2e.sh <worktree-name> --video

Expected success output — a POD-E2E SUMMARY ending like:

  ✅ auth — GET /api/sessions → 200 with token, 403 without
  ✅ playwright — headless chromium loaded dashboard …
result:       2 passed, 0 failed
ARTIFACT_DIR=~/.kirocrew-pods/.e2e-artifacts/<worktree-name>

Exit code = number of failed phases (0 = all green) once phases run. Setup errors exit before any phase with their own codes (see the flag table below); a production-port SAFETY refusal exits 1. Then look at the evidence: Read the screenshots in that ARTIFACT_DIR (fe-smoke.png, plus any spec screenshots) to confirm the real UI rendered — not a 403/blank page.

To smoke-test isolation after it finishes:

kirocrew pod ls          # should be empty (torn down)
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5476/api/sessions   # live plane still alive

The interface — pod tools if you are an agent, the CLI if you are a human

If you are an agent session, reach for the pod_up / pod_down / pod_status / pod_ls MCP tools before a shell. A pod is a systemd --user unit, so every pod verb needs the systemd user bus. Whether your shell can reach that bus depends on the sandbox your session runs behind: under Kiro Crew's own sandbox it can (see security.md, "Scoped user-bus locator forward"), but behind an outer sandbox with its own user namespace it cannot, and kirocrew pod up then fails with Permission denied. systemctl --user is-system-running tells you which case you are in. The tools work either way -- the gateway holds the host bus and does the systemd part -- so they are the portable choice, and the only one on a host that denies you the bus.

pod_up     {"worktree": "<wt>"}   -> prose: port, base_url, token (+ its two limits)
pod_status {"worktree": "<wt>"}   -> status + port + health
pod_ls     {}                     -> every pod active on this host
pod_down   {"worktree": "<wt>"}   -> stopped, HOME reclaimed

Provisioning is not one of them: a cold venv plus an SPA build is minutes of work, and a single blocking tool call for that dies to any timeout with no way to learn the outcome. An unbuilt worktree is refused with the CLI's own remedy in the message; run kirocrew pod provision <wt> or use the Dev Fleet page's Provision button, which streams its output.

Everything after step 1 below -- curl against the handle, drive the SPA with Playwright, read the screenshots -- is unaffected: those are ordinary loopback HTTP, which a sandbox does not block. Only the systemd control path is walled off.

The tools need the Dev Fleet app enabled; a disabled app refuses with app_not_enabled.

The CLI is the same operations for a human at a terminal:

# 1. bring the pod up, get a handle (JSON: base_url + token + port)
kirocrew pod up <wt> --json
#    → {"name":"<wt>","status":"up","port":7958,
#       "base_url":"http://127.0.0.1:7958","token":"…","ttl":"2h"}

# 2. test against the handle — full stack, ONE port:
curl -s "$base_url/api/<anything>?token=$token"      # backend API
#    open  $base_url/?token=$token  in Playwright     # frontend SPA (same port)

# 3. destroy it — deletes the HOME and verifies it is gone (nonzero if not),
#    live gateway untouched
kirocrew pod down <wt>

The token is a link with two limits. The pod accepts the raw token for 5 minutes, and the first request that carries it opens a session, kept in a cookie, that lasts 2h (ttl in the JSON is that session lifetime). So use the token straight away and drive the pod with the session: keep curl's cookie jar from that first request (curl -c jar -b jar ...) or let Playwright keep its context. A ?token= sent after the window is refused with 403 token expired; pod_up (or kirocrew pod token <wt>) on the running pod leaves it running and returns a fresh token.

Other verbs: ls (list running pods) · status <wt> · token <wt> · url <wt> · logs <wt> · provision <wt>. Run kirocrew pod --help for the full list. token, url, logs and provision have no tool of their own yet: pod_up already returns the token and url, and pod logs stay a human-facing read.

Isolation guarantees (enforced by the pod runtime): own KIROCREW_HOME, own port, no tunnel (can't grab the real Slack identity), --no-crons, and cleanup on pod down. A pod can never collide with the live gateway and many can run at once.

Resource ceilings are Linux-only. On Linux the unit sets cgroup MemoryMax=4G / CPUQuota=200%, which the kernel enforces. On macOS there is no such ceiling and none is emitted — macOS has no cgroups, and nothing it does offer bounds the total memory of a process tree or hard-caps CPU (RLIMIT_AS covers one process's address space, not resident memory, and the gateway spawns agent subprocesses that each get their own limit). So on a Mac a runaway pod can starve the machine; every other isolation property above still holds.

Teardown belongs to pod down, on both platforms. It stops the service, waits for the process tree to drain, deletes the isolated HOME, and verifies the directory is gone — a HOME that survives is reported as a failure, never as zero residue. Nothing reclaims from a post-stop service hook: systemd would run one before the final kill of the pod's cgroup (racing the pod's own subprocesses) and on the stop half of a restart. So a pod that goes away WITHOUT a down (host crash, force-reboot, a raw systemctl --user stop) leaves its HOME behind on either OS; pod ls reports those (reclaim each with pod down <name>).

This is also what makes a seeded pod home survive systemctl --user restart and Restart=on-failure instead of silently reverting to a blank instance.

Bundled suite (quick path)

bash <app-skills-dir>/pod-e2e/scripts/pod-e2e.sh <worktree-name>

Runs the bundled orchestrator end-to-end. Prints a POD-E2E SUMMARY ending in ARTIFACT_DIR=<path> and exits with the number of failed phases (0 = all green). Before any phase runs it exits with a setup code instead:

exit	meaning
64	bad arguments, bad worktree name, or unusable --handle-json
65	no kirocrew CLI with pod on PATH, or the artifact dir escapes .e2e-artifacts
66	the worktree checkout could not be resolved
67	no usable CLI in the worktree venv (kirocrew pod provision <wt> --venv-only)
70	the live-plane port could not be resolved (python3 missing)
1	SAFETY refusal: the pod resolved to the production port (also a 1-failure run; read the summary)

Flags:

flag	effect
--handle-json <path>	run against a pod the CALLER started, from the handle in that file (see below)
--keep / --no-stop	leave the pod running after tests (debug)
--api-only	skip the Playwright phase (leaves a boot + auth check)
--fe-only	accepted no-op — no test-suite phase exists to skip
--video	record the session at 1080p → .webm + .mp4 (finalization is time-capped)
--no-suppress-first-run	let first-run modals appear (onboarding tests only)

Running the suite when you cannot reach the bus

The script drives the pod with kirocrew pod verbs, and those need the systemd user bus, so on a host whose outer sandbox denies it the harness cannot start -- even though every phase after the first is ordinary loopback HTTP. --handle-json is that path: boot the pod with the pod_up tool, then build the JSON object shown below out of the values it reports and hand that file over. Do not paste the tool's own reply into the file: it is labelled prose, and json.load rejects it with exit 64.

# pod_up {"worktree": "<wt>"} -> {"name": "<wt>", "base_url": ..., "token": ..., "port": ...}
# pod_status {"worktree": "<wt>"} -> health
# keep pod_up's name and write the fresh health code into that object, then:
bash <app-skills-dir>/pod-e2e/scripts/pod-e2e.sh <wt> --handle-json "$H/handle.json" --video
# ... and when you are done, pod_down {"worktree": "<wt>"}

The file is a JSON object you build: name is the <wt> you passed to pod_up (its reply names it but carries no name field), base_url, token and port are the values its reply prints, and health is the code pod_status reports:

{"name": "<wt>", "base_url": "http://127.0.0.1:7813", "token": "...", "port": 7813, "health": 200}

name must be a non-empty string exactly equal to the <wt> argument. The harness refuses a missing, non-string, empty, or different name before any phase runs; it never normalizes this field. Use exactly the worktree string you passed to pod_up. base_url must be an HTTP loopback address carrying an explicit port that is none of: the configured live-plane port (KIROCREW_POD_LIVE_PORT), its default 5476, and the reserved 7777. The default stays refused even when the variable names another port, because that variable is read from the harness's own environment: a deployment that sets it only in the gateway's unit would otherwise leave 5476 open here. Export it in the shell that runs the harness so the configured port is refused too. It and token may contain no whitespace or control characters. token must use the generated-token alphabet (A-Z, a-z, 0-9, -_.~+/=), and port must be a canonical decimal integer from 1 through 65535 that matches the URL port. The harness validates these fields once, then runs from a canonical payload: base_url is rebuilt as http://<loopback-host>:<port> with no path, port is plain decimal, and the token is the validated value. No raw handle field reaches the production-port refusal, curl configuration, or Playwright. That is not shape pedantry: validating one representation and using another can point the harness at the live plane. The pod_up producer already emits the canonical URL, token, and port; add only the fresh health code from pod_status.

In that mode the script calls no pod verb at all, and three things follow:

• The pod is yours, not the harness's. It is never stopped on exit, whether the run passes or crashes. Call pod_down yourself.
• The health verdict is read, not measured. The summary says health — supplied by the caller, and a stale handle means the run is judging a pod that has already died. Re-read it with pod_status immediately before the run, and pass what it reports rather than a remembered 200.
• The lifecycle ran on the gateway's build, not the worktree's. The CLI path deliberately pins the worktree's own kirocrew so pod up / pod down exercise the branch under test. Nothing pins the gateway. So for a diff that changes pod lifecycle code itself, the CLI path is the one that tests it -- a green handle-mode run says the branch's gateway boots and renders, not that the branch's pod code works.
• name ties the handle to a worktree, not to a clone. The check refuses a handle whose name is not the worktree you asked for, but a pod name is a worktree basename, and two clones on one host can each hold a worktree of that name. Run the harness from the clone whose pod you booted: with a handle from the other one, the specs and the artifact directory come from here while the requests go there, and the verdict is filed under the wrong checkout. Nothing in the handle, and nothing the pod serves, identifies a checkout today -- closing it needs pod_up to emit a machine-readable handle that carries one.

A missing or malformed handle is refused before any phase runs, naming the field that is wrong.

It does not run the worktree's test suite. There is no pytest phase, on purpose: the suite is large and none of it needs a pod, CI runs it on the merge ref, and a full local fan-out on a shared box costs far more than the browser check this harness exists for. Run the tests your change actually touches yourself, in the worktree (see the kirocrew-worktree-dev skill), and let CI own the full suite.

What each phase does

1. up — kirocrew pod up <wt> --json. If already active, reuses it (and won't stop it on exit). With --handle-json nothing is started at all: the handle IS this phase. Boots the worktree's own gateway with --no-crons, blank-seed DB, isolated HOME. If this phase fails with gateway still starting after Ns in pod-up.log, the gateway was alive but slower than the health-wait budget (default 90s): re-run with KIROCREW_POD_HEALTH_SECS=<higher> exported — the harness passes its environment through to the pod up it spawns.
2. health — polls kirocrew pod status <wt> --json until its health is 200/401/403 (≤60s). Deliberately not a bare curl base_url/api/health: a derived port is routinely held by another pod or by the live gateway, every gateway answers that path identically, so a 200 there proves only that something is listening. pod status reports the pod's OWN health and returns -2 when the responder is provably somebody else's, which this phase reports as a port conflict naming PORT=. On timeout it dumps logs to boot-fail.log and aborts.
3. auth — proves auth: /api/sessions → 200 with token, 403 without. Token comes from kirocrew pod up --json output (no manual minting needed).
4. Playwright — pod-playwright.py (run with a Playwright venv + bundled chromium) loads /?token= headless, asserts the SPA rendered (screenshots fe-smoke.png), then exec's the optional PLAYWRIGHT_SPEC with a live authed page in scope.
   • If the Playwright interpreter is not executable, the FE phase fails — it does not skip. A run that captured zero screenshots must never report a green summary. --api-only is the one clean skip.
   • --video requires ffmpeg for mp4 transcoding; if absent, the .webm is kept but no .mp4 is produced.
   • Bounded, always. The whole phase runs under timeout (POD_E2E_PW_TIMEOUT, default 600s) and each browser-teardown step under its own cap (POD_E2E_TEARDOWN_TIMEOUT, default 30s). Video finalization (context.close()) has been observed to block forever after a spec passed; on expiry the driver keeps every artifact, kills the browser tree, and exits, and the summary reports playwright — TIMED OUT as a distinct outcome. A recording that grew past 200MB is reported and left un-transcoded — for a short spec that size is itself a defect signal. The per-step cap uses SIGALRM, so on a platform without it the teardown degrades to unbounded and says so in the log (the harness is POSIX-only anyway); the phase-level timeout still applies.
5. collect — all logs + screenshots land in ~/.kirocrew-pods/.e2e-artifacts/<wt>/. The Playwright driver appends its rows (smoke, spec, spec record(...) rows, teardown) to verdict.jsonl as they are decided; the up/health/auth verdicts appear only in the printed summary (and playwright.log is unbuffered), so a stalled or killed run still leaves a readable verdict. The file is truncated at the start of every run — including runs that skip the FE phase — so it can never show a previous run's rows. The rest of the artifact dir DOES persist across runs, so check timestamps before trusting an old screenshot.
6. stop — kirocrew pod down <wt>: stops the service, waits for its process tree to drain, deletes the isolated HOME, and verifies it is gone — a HOME that survives fails the command rather than being reported as zero residue. Skipped if --keep or if the pod was already up.

The test manifest (.pod-test.sh)

Optional per-worktree file declaring how THIS feature is tested. Searched at <worktree>/.pod-test.sh then <worktree>/src/kiro_crew/.pod-test.sh. The manifest is parsed declaratively (a PLAYWRIGHT_SPEC= line is extracted textually) — it is never sourced or eval'd on the host.

# .pod-test.sh
PLAYWRIGHT_SPEC=".pod-e2e/feature.spec.py" # frontend spec, relative to the manifest's dir

The resolved spec must live under <worktree>/.pod-e2e/: the driver refuses any other path ("outside allowed directories") and fails the run. A manifest under src/kiro_crew/ therefore needs PLAYWRIGHT_SPEC="../../.pod-e2e/<spec>.py" or an absolute path. The spec runs only when the smoke phase passed.

Trust model

The pod isolates the gateway under test (own KIROCREW_HOME, own port, no tunnel, resource caps). The harness is not a sandbox: it runs the worktree's own gateway, and exec's that branch's PLAYWRIGHT_SPEC, as your user — exactly like building and running the checkout yourself. Only run pod-e2e against branches you would be willing to build and run locally.

Playwright spec contract

A spec is plain Python exec'd with these names in scope (no imports needed): page (already on the authed app), context, base_url, token, artifact_dir, expect (Playwright's native web-first assertion — expect(locator).to_be_visible(), auto-retries), expect_true(cond, msg) (boolean fallback, raises AssertionError), and record(name, ok, detail="") (append a per-assertion row to verdict.jsonl immediately, so a later stall still leaves your decided results on disk — an ok=False row fails the run, it is not a silent note). Assert UI, take screenshots into artifact_dir. Run with --video to also record a .webm (+ a shareable .mp4) at 1080p, paced.

First-run noise is auto-suppressed

A fresh pod = a real first-run: every Playwright context starts with empty localStorage, so onboarding/changelog modals would pop up and overlay the feature you're testing. The runner handles this automatically:

• pre-seeds localStorage so theme modal never mounts;
• dismisses any modal that still appears by pressing Escape + clicking [aria-label="Close"] if present.

Pass --no-suppress-first-run to let those modals appear (only if testing the onboarding flow itself).

PRIMARY USE — dev agent delegates to a QA agent

When you (the agent building a feature) want it tested, spawn a separate QA agent via spawn_run and hand it the worktree name. You keep coding; the QA agent runs the isolated pod, inspects the evidence, triages failures, and reports a verdict back as a completion event.

The QA agent prompt (copy, fill <wt> + the feature one-liner)

spawn_run(task="""
You are a QA engineer verifying the KiroCrew feature in worktree '<wt>'.
Feature under test: <one-line description of what this branch adds>.

Run the isolated end-to-end suite (it spins a throwaway pod on its own port,
never touches the live instance, and tears it down after):

    bash <app-skills-dir>/pod-e2e/scripts/pod-e2e.sh <wt> --video

Rules:
- Do NOT `cat` any .local_secret yourself (credential-read blocked). The script
  mints the token internally via the CLI — just run the one command above.
- If that command dies with a `Permission denied` from systemd, your sandbox
  denies you the user bus. Do NOT hand-roll curl and Playwright in its place:
  boot the pod with the `pod_up` tool, build one JSON file holding its `name`,
  `base_url`, `token` and `port` plus the `health` from `pod_status`, re-run the
  same command with `--handle-json <file>`, and call `pod_down` when the verdict
  is written. Every one of those fields is required and `name` must equal `<wt>`,
  so a file missing one is refused with exit 64 before any phase runs.
- After it finishes, READ the artifacts in the printed ARTIFACT_DIR:
  verdict.jsonl (Playwright rows, written as decided — trust this even if the
  run was killed; up/health/auth are only in the printed summary),
  playwright.log, fe-*.png screenshots (use the Read tool on the .png to
  actually look at the UI), and boot-fail.log if present.
- Before you run, record what you are testing: `git -C <worktree> rev-parse HEAD`
  and `git -C <worktree> status --porcelain`. The dev keeps coding in the same
  worktree and the pod pins a path, not a commit.

Then return a QA VERDICT, not a raw dump:
  1. Overall: PASS / FAIL / BLOCKED (couldn't even boot the pod).
  2. Per check (auth / playwright): pass|fail + one-line evidence.
  3. For each FAIL: triage it — is it (a) a real regression in the feature,
     (b) a flaky/timing issue, or (c) an environment problem (missing venv,
     missing dist, port clash)? Cite the log line or screenshot that proves it.
  4. The ARTIFACT_DIR path so the dev can open screenshots/video.
  5. The HEAD SHA and dirty state you recorded before the run.
""")

Cite a PASS only for the SHA the verdict names. A later commit needs a new run.

One sub-agent is right here: Playwright output, videos and pod logs would flood your context, and only the verdict is needed back.

When to delegate vs run inline

• Delegate to a QA agent when the suite is long (Playwright + video) and its raw output would swamp your context -- that is the bulk-output reason to spawn.
• Run inline yourself (default for a quick smoke): you want the result in your own turn, and "not derailing my context" is not a reason to delegate.

Parallel QA across branches: spawn one QA agent per worktree in a single spawn_run tasks=[...] call — each pod gets its own port and isolated HOME, so they don't collide.

Prerequisites & the provisioning on-ramp

A worktree must be built before it can be podded — its own .venv/bin/kirocrew (editable install) + a built SPA bundle (static/dist). The pod boot refuses without them.

kirocrew pod up <wt>                 # auto-builds the venv; FAILS LOUD if no dist
kirocrew pod up <wt> --provision     # full on-ramp: venv + build, then up
kirocrew pod provision <wt>          # just the on-ramp (venv + dist)
kirocrew pod provision <wt> --venv-only

Every failure teaches the next step: no worktree → create one; no venv → auto; no dist → build-or---provision.

• Playwright venv: controlled by env KIROCREW_PW_PY. If that interpreter is missing or not executable, the FE phase fails — it does not skip. A run that captured zero screenshots must never report a green summary. Set it up once, pinning the version that matches the chromium build already on disk:

  python3 -m venv <path> && <path>/bin/pip install playwright==1.61.0
  export KIROCREW_PW_PY=<path>/bin/python
  

  To skip the frontend phase deliberately, pass --api-only — that is the only clean skip.

• --video needs ffmpeg on PATH (or pointed to by POD_E2E_FFMPEG env). If absent, .webm is kept but no .mp4 transcoding occurs. Recording finalization is time-capped (see POD_E2E_TEARDOWN_TIMEOUT), so --video can cost you the .mp4 — never the verdict.

Hands off the live plane

This skill only ever talks to pod ports (default KIROCREW_POD_BASE_PORT 7810 + 1..199, or a pinned PORT=). It must never restart or touch the live gateway. If the derived port ever resolves to the production port the orchestrator refuses and exits.

Attach approved QA media to the PR (MANDATORY workflow)

QA screenshots and demo videos follow a review-then-attach contract:

1. Deliver to the user first. Send the screenshots/video for review (after the usual frame inspection for overlays -- first-run modals, toasts, theme pickers). Never attach media the user has not seen.
2. Wait for explicit approval of the media. A silent user is NOT approval.
3. On approval, attach to the PR automatically -- do NOT ask again. The media is a GitHub attachment, never a commit: nothing is copied into the worktree, nothing is amended, nothing is pushed.
   • Write the PR body to a scratch file outside the worktree and reference the approved files by their local paths: ![Settings page, empty state](<ARTIFACT_DIR>/fe-settings.png). Put the 2-3 most telling shots inline and fold the rest into <details>. A video MUST stand alone in its own paragraph -- ![](<ARTIFACT_DIR>/demo.mp4) with a blank line above and below -- to render as an inline player; inside a sentence it renders as a link.
   • Run gh pr edit <n> --body-file <body> --attach <ARTIFACT_DIR>/fe-settings.png --attach <ARTIFACT_DIR>/demo.mp4 (one --attach per file; gh >= 2.99, check gh --version). Each referenced path is rewritten in place to a permanent https://github.com/user-attachments/assets/<uuid> URL; an attached file the body does not reference is appended at the end. Limits: 10 MB per image/GIF, 100 MB per video. --attach needs push access to the repository (its upload endpoint 404s on read permission, cli/cli#14302) -- a fork contributor without it either drags the file into the description in the web UI, which yields the same URL, or commits it with git add -f temp-screenshots/<topic>/shot.png and references that repository-relative path; the review lanes read both.
   • Verify the body update landed: gh api repos/<o>/<r>/pulls/<n> --jq .body | grep -c user-attachments prints the number of files you attached.
   • The URL is tied to no commit or branch, so a later amend, force-push, branch deletion or the merge leaves it valid; nothing is ever re-pinned.
4. Attach before asking for PR approval. The evidence goes in as a body edit -- no push, no approval reset -- and it must be in the body before the user is asked to approve the PR, so the approval covers what a reviewer sees.

Keep the rest of the tree clean

The e2e suite already writes its logs and screenshots to ~/.kirocrew-pods/.e2e-artifacts/<wt>/ -- outside the worktree -- by design; don't copy those raw logs back into the worktree "to keep them with the branch." No QA output belongs in the tree: approved media is uploaded from ARTIFACT_DIR as a PR attachment (above), and everything else -- raw *.log dumps, extra frames, scratch notes, the .pr-body.md you fed to gh pr create -- stays outside (write it under a mktemp -d). Before ending the session, git status --porcelain must be empty: a dirty tree fail-closes Dev Fleet's "Prune merged" (merged_dirty) so the merged worktree can't be reaped. See the kirocrew-worktree-dev skill, "Rule 9 -- Leave the worktree clean (so prune can reap it)."