automate-review-process
Design a risk-tiered review process for code, design, or approvals. Measures review latency, splits the work between checks, AI reviewers, and humans, and produces the policy plus starter config. Use
SKILL.md.claude/skills/automate-review-process/SKILL.md
GitHubAutomate the Review Process
Help an engineering team spend human review time only where judgment matters. Everything deterministic moves to checks, and everything else is routed by risk.
Reasoning Framework
"Fully automated review" is the wrong goal. Review is a risk-control system, so design it like one: tiers of risk, gates matched to each tier, evidence that each gate ran, an escape hatch for emergencies, and metrics that show whether it works. Automate what a script can decide (format, lint, tests, secrets, size, forbidden files). Let an AI reviewer take the first pass on logic. Keep human approval for design intent, correctness in risky areas, and anything compliance requires.
Output Contract
| Artifact | Format | Handed to |
|---|---|---|
| Review baseline | Markdown table from scripts/pr_review_stats.py |
Eng lead, PM |
| Review process design | Markdown, from references/review-policy-template.md |
Eng leads, team |
| Starter config | CODEOWNERS, PR template, branch-protection checklist | Repo admins |
When to Use
- PRs wait days for review, or reviews are rubber stamps
- Review rules differ by reviewer and nobody knows the policy
- Adding an AI code reviewer and deciding what it may block or approve
- Design reviews or launch approvals that stall
When NOT to Use
- Reviewing a specific PR: use your code-review tool (for example
/code-reviewin Claude Code) - Launch go/no-go for one release: use
launch-checklist
Inputs
- Scope: code review, design review, approvals, or all three.
- Repos and access: GitHub with
ghauthenticated gives a measured baseline; other hosts need the same numbers exported by hand. - Team: size, roles, time zones, who owns which areas.
- Pain: slow, inconsistent, missed defects, reviewer burnout, or something else.
- Constraints: compliance rules (for example two-person approval), regulated or sensitive code paths.
- Tools available: CI, AI reviewers, merge queue.
Steps
Measure the baseline (script).
gh pr list --state merged --limit 200 \ --json number,createdAt,mergedAt,additions,deletions,author,reviews > prs.json python3 scripts/pr_review_stats.py prs.json -o review-baseline.md python3 scripts/pr_review_stats.py prs.json --exclude-bots # human-only viewBefore you conclude anything, ask how review actually happens. Reviews outside GitHub's review feature do not appear in these numbers: pre-PR AI gates, verdicts stamped in the PR body, pairing. In that case "merged unreviewed" is a measurement gap, not a finding.
Map the flow. Author → automated checks → reviewer assignment → rounds → approval → merge → deploy. Separate waiting time (queue) from rework time (rounds). Note the failure modes: oversized PRs, unclear ownership, nit rounds, late design feedback.
Define risk tiers by path. This is the core of the design.
- Tier 0: docs, copy, tests only. Automated checks, optional async review.
- Tier 1: standard product code. Checks, an AI first pass, one human reviewer.
- Tier 2: auth, payments, data migrations, infra and permissions, secrets, public APIs. Checks, AI review, and a code-owner approval, with no auto-merge.
Write tiers as an allowlist of low-risk paths. Any path the rules don't recognize falls into the stricter tier.
Automate the deterministic layer. Formatting, lint, type checks, tests, secret scanning, dependency review, a PR size warning (about 400 changed lines), forbidden-file checks, and PR template completeness. If a script can decide it, it is not a reviewer's job.
Design the AI review gate (if used):
- Where it runs: locally before the PR opens (fast, cheap) or as a CI check (enforceable).
- What blocks: findings at or above an agreed severity.
- The record: every verdict is written somewhere verifiable (a check run or a stamp in the PR body). Nobody writes "AI reviewed" without that artifact.
- A round cap (for example 3). After that, a human decides instead of grinding through more rounds.
- A skip protocol for when the reviewer is down or out of quota. Only Tier 0 and Tier 1 changes may skip, and each skip records the reason and the substitute review.
- Cost routing: a cheaper model or lower effort for Tier 0 surfaces, the strongest for Tier 2.
Set human review rules. A response-time SLA (for example first response within one business day), assignment (CODEOWNERS plus rotation), what humans review (intent, design, risk; not style), merge authority per tier, and an emergency path (hotfix now, review within 24 hours).
Cover design reviews and approvals if in scope. Apply the same tiering (new pattern or flow versus copy tweak) and use async written review with a decision deadline. Route launch approvals through
launch-checklist.Plan the rollout. Pilot on one repo or team for 2–4 weeks, rerun the script, and compare against the baseline. Target metrics: first-review p50/p90, merge p50, share of PRs over 400 lines, change-request rounds, and escaped defects per tier.
Write the design from
references/review-policy-template.md. Keep the policy to about one page.
Examples
- "Our PRs sit for two days." Measure, find that XL PRs dominate the p90, and propose a size budget, CODEOWNERS routing, and an AI first pass for Tier 1.
- "We want an AI reviewer to block merges." Define what it blocks on, where verdicts are recorded, the round cap, and the skip protocol, then pilot on Tier 0 and Tier 1 only.
Guardrails
- Never remove a human approval that compliance or policy requires.
- No auto-merge for Tier 2 paths.
- Every rule states its reason. If a rule can't be explained, cut it.
- "Reviewed" means a verifiable artifact exists, for humans and AI alike.
- Measure before and after. Don't declare success on anecdotes.
- Keep private repo names, internal paths, and team names out of anything committed to this public repo.
Related Skills
devex-survey: developer sentiment on review turnaroundlaunch-checklist: approval gates for releasesdecision-brief: get sign-off on the new policyengineering-foundations: review drills and habitslibrary/lenny-podcast/lenny-running-design-reviews/SKILL.md: design review practice