bulwark-verify

Generate runnable verification scripts for components. Orchestrates assertion-patterns and component-patterns to produce executable scripts.

SKILL.md

GitHub

Bulwark Verify

Purpose

Generate runnable verification scripts that test real component behavior without mocks. This skill orchestrates assertion-patterns (P2.1) and component-patterns (P2.2) to produce executable scripts for any component type.

When to Use

Load this skill when:

  • User requests /bulwark-verify [path]
  • test-audit Step 7 needs verification scripts
  • Generating standalone verification for a component

DO NOT use for:

  • Running existing tests (use just test)
  • Writing unit tests (implement directly)
  • Test auditing (use test-audit skill)

Usage

/bulwark-verify [path] [--execute]

Examples:

  • /bulwark-verify src/cli.ts - Generate verification script for CLI
  • /bulwark-verify src/server.ts --execute - Generate and run
  • /bulwark-verify - Infer from recent context

MANDATORY EXECUTION STEPS

WARNING: These steps are BINDING instructions, not suggestions. You MUST execute each step in order. Do NOT substitute your judgment for these instructions. Do NOT skip sub-agent spawning. Do NOT modify the execution flow.

When this skill is loaded, execute these steps exactly as written:

Step 1: Resolve Target

IF $ARGUMENTS provided:
    target = first non-flag argument
    execute_flag = "--execute" in $ARGUMENTS
ELSE:
    Look for component files in recent conversation context
    IF found: target = that path
    ELSE: Ask user: "Which component should I generate a verification script for?"

Step 2: Detect Project Language

Check for project manifest files in order (search from target file's directory up to project root):

Check Language Test Runner
package.json exists Node jest/vitest/node
pyproject.toml OR setup.py exists Python pytest/python
Cargo.toml exists Rust cargo test
None of the above Generic bash

Step 3: Analyze Component

  1. Read the target file

  2. Identify component type using indicators from component-patterns skill:

    • Has spawn/exec/execSync imports → Process Spawner
    • Has listen()/createServer/express()/fastify() → HTTP Server
    • Has fs.readFile/parse functions → File Parser
    • Has process.argv/yargs/commander/argparse → CLI Command
    • Has database imports (pg, mysql, mongoose, prisma) → Database
    • Has fetch/axios/got/requests calls → External API
  3. Load dependent skills:

    • Load assertion-patterns skill content
    • Load component-patterns skill content
  4. Load bug-magnet-data for the component type [REQUIRED]:

    • Read the context file matching the component type:
      Component Type Context File
      CLI Command bug-magnet-data/context/cli-args.md
      HTTP Server bug-magnet-data/context/http-body.md
      File Parser bug-magnet-data/context/file-contents.md
      Database bug-magnet-data/context/db-query.md
      Process Spawner bug-magnet-data/context/process-spawn.md
      External API bug-magnet-data/context/http-body.md
    • Read the "Applicable Categories" section from the context file
    • Load T0 + T1 data files listed in the context file:
      • T0 (Always): data/strings/boundaries.yaml, data/numbers/boundaries.yaml
      • T1 (Common): Based on context file recommendations
  5. Select applicable patterns:

    • From assertion-patterns: Identify T1-T4 transformation patterns relevant to the component
    • From component-patterns: Select the matching component type template
    • From bug-magnet-data: Extract concrete edge case values to include in generated script

Step 4: Generate Script [SPAWN-REQUIRED]

You MUST spawn a Sonnet sub-agent for script generation. Do NOT generate the script yourself.

Task(
    description="Generate verification script for {component_name}",
    subagent_type="general-purpose",
    prompt=<constructed_4part_prompt_from_template_below>
)

The sub-agent writes the script to tmp/verification/{component_name}-verify.{ext}.

Step 5: Validate Generated Script [REQUIRED]

After the sub-agent returns, validate the generated script syntax:

Language Validation Command Success
Node node --check {script_path} Exit 0
Python python -m py_compile {script_path} Exit 0
Bash bash -n {script_path} Exit 0

If validation fails:

  1. Read the error message
  2. Fix the syntax issue in the generated script
  3. Re-validate until successful
  4. Only then proceed to Step 6

Step 6: Report Results

Present summary to user:

## Verification Script Generated

**Component:** {component_path}
**Type:** {component_type}
**Language:** {language}

**Script location:** tmp/verification/{name}-verify.{ext}

**To run manually:**

{runner_command}


If --execute flag was provided:

  1. Run the generated script using Bash
  2. Capture output
  3. Report PASS/FAIL counts
  4. Show any failures with details

Generation Prompt Template

Use this 4-part prompt when spawning the Sonnet sub-agent:

## GOAL

Generate an executable verification script for `{component_path}` that tests real
component behavior without mocks. The script must verify observable output and
report clear PASS/FAIL for each test.

## CONSTRAINTS

- Language: {detected_language}
- Test runner: {runner} (e.g., jest, pytest, bash)
- Component type: {detected_type}
- MUST be directly executable: `{runner_command}`
- MUST use assertion patterns from assertion-patterns skill (real output, not mock calls)
- MUST follow component pattern from component-patterns skill ({component_type} verification)
- MUST include edge cases from bug-magnet-data (boundaries, special values, injection patterns)
- Include setup and teardown if component requires it
- Report clear PASS/FAIL for each verification
- Handle cleanup on both success and failure (use trap for bash, afterAll for jest, fixtures for pytest)
- Exit with code 0 on all pass, code 1 on any failure
- EXCLUDE destructive patterns marked `safe_for_automation: false` (add as commented-out manual tests)

## CONTEXT

### Component Code
```{language}
{component_content}

Component Type

{detected_type}

Applicable Assertion Patterns (from assertion-patterns)

{relevant_assertion_patterns}

Applicable Component Pattern (from component-patterns)

{component_pattern_template}

Edge Cases (from bug-magnet-data) [REQUIRED]

Include these edge cases in verification tests:

T0 (Always include): {t0_edge_cases_from_bug_magnet_data}

T1 (Include if component handles input): {t1_edge_cases_from_bug_magnet_data}

Destructive patterns (manual-only - add as comments): {destructive_patterns_if_any}

OUTPUT

Write script to: tmp/verification/{component_name}-verify.{ext}

Extension mapping:

  • Node → .test.js
  • Python → _test.py
  • Rust → .rs (or .sh if cargo test not suitable)
  • Generic → .sh

Script Structure

  1. Setup (create temp files, start services, initialize test DB)
  2. Execute component under test
  3. Verify observable output (not mock calls)
  4. Report PASS/FAIL clearly for each test
  5. Cleanup (kill processes, remove temp files)
  6. Exit with appropriate code (0 = all pass, 1 = any fail)

Report your actions to the log file

Write to: logs/bulwark-verify-{YYYYMMDD-HHMMSS}.yaml


---

## Output Formats

### Generated Script Location

tmp/verification/{component-name}-verify.{ext}


### README Files (Per-Component)

If generating a README for the verification script, name it per-component to avoid overwrites:

tmp/verification/{component-name}-README.md


**NOT:** `tmp/verification/README.md` (would be overwritten by subsequent runs)

### Cleanup Behavior

Generated scripts **persist in `tmp/verification/`** for inspection and debugging:
- Scripts are NOT automatically deleted after execution
- `tmp/` is in `.gitignore` - scripts won't be committed
- Manual cleanup: `rm -rf tmp/verification/*`

This allows:
- Post-run inspection of generated scripts
- Iterative refinement of verification approach
- Debugging when tests fail

### Log Schema
```yaml
# Top-level — required for Stop-hook per-file pipeline-recursion suppression.
# List every .sh verification script generated for this run (Bulwark verify
# scripts live under tmp/verification/ but are SCRIPT-bucket files for
# coverage purposes). Paths relative to ${CLAUDE_PROJECT_DIR}. Empty list
# `[]` if no script was emitted. Missing field disables suppression.
reviewed_files:
  - tmp/verification/{component-name}-verify.sh

metadata:
  skill: bulwark-verify
  timestamp: {ISO-8601}
  model: sonnet

generation:
  target: {component_path}
  language: node|python|rust|generic
  component_type: cli|http|file-parser|process|database|api
  script_path: tmp/verification/{name}-verify.{ext}
  patterns_used:
    assertion: [T1_transformation, T2_transformation]
    component: "{component_type} verification"

execution:  # Only if --execute
  ran: true
  runner: {runner_command}
  exit_code: 0|1
  duration_ms: 1234
  results:
    pass: 3
    fail: 0
  output: |
    === Verification: {component} ===
    Test 1: Basic functionality... PASS
    Test 2: Error handling... PASS
    Test 3: Edge cases... PASS
    === All tests passed ===

summary: |
  Generated verification script for {component} ({type}).
  Script: tmp/verification/{name}-verify.{ext}
  Run with: {runner_command}
  [Execution: 3 passed, 0 failed]

Diagnostic Schema

# Top-level — mirror the same list emitted in the run log (Stop hook contract).
reviewed_files:
  - tmp/verification/{component-name}-verify.sh

skill: bulwark-verify
timestamp: {ISO-8601}
diagnostics:
  model_requested: sonnet
  model_actual: sonnet
  context_type: main
  language_detected: node|python|rust|generic
  component_type: cli|http|file-parser|process|database|api
  patterns_loaded:
    - assertion-patterns
    - component-patterns
  script_generated: true
  script_path: tmp/verification/{name}-verify.{ext}
  execution_requested: true|false
  execution_result: pass|fail|skipped
  completion_status: success|error

Write diagnostic output to: logs/diagnostics/bulwark-verify-{YYYYMMDD-HHMMSS}.yaml


Integration with test-audit

When test-audit Step 7 invokes this skill:

  1. test-audit provides the test file path and violation info
  2. This skill generates a verification script as intermediate artifact
  3. The script validates the rewrite approach before modifying the test
  4. If verification passes, test-audit proceeds with the rewrite

Flow:

test-audit Step 7
    → Load assertion-patterns
    → Load component-patterns
    → Generate verification script (tmp/verification/)
    → Run verification script
    → If pass: Apply rewrite to test file
    → If fail: Report issue, do not rewrite

Runner Commands by Language

Language Default Runner Command
Node node (built-in test) node --test tmp/verification/{name}-verify.test.js
Node (Jest) jest npx jest tmp/verification/{name}-verify.test.js
Python pytest pytest tmp/verification/{name}_test.py -v
Rust cargo cargo test --test {name}
Generic bash bash tmp/verification/{name}-verify.sh

When generating scripts, prefer the simplest runner:

  • For Node: Use built-in node --test if no external deps needed
  • For Python: Use pytest for better output
  • For Generic: Always use bash

Error Handling

Target file not found

Error: Target file not found: {path}
Please provide a valid path to the component you want to verify.

Language detection failed

Warning: Could not detect project language. Using generic (bash) templates.
To specify: /bulwark-verify {path} --lang=node

Component type detection failed

Warning: Could not determine component type from code analysis.
Please specify: /bulwark-verify {path} --type=cli

Available types: cli, http, file-parser, process, database, api

Script generation failed

Error: Failed to generate verification script.
Reason: {error_message}

Check logs/bulwark-verify-*.yaml for details.

Examples

Example 1: CLI Component

User: /bulwark-verify src/cli.ts

Output:
## Verification Script Generated

**Component:** src/cli.ts
**Type:** CLI Command
**Language:** Node

**Script location:** tmp/verification/cli-verify.test.js

**To run manually:**

node --test tmp/verification/cli-verify.test.js


Example 2: HTTP Server with Execution

User: /bulwark-verify src/server.ts --execute

Output:
## Verification Script Generated

**Component:** src/server.ts
**Type:** HTTP Server
**Language:** Node

**Script location:** tmp/verification/server-verify.test.js

**Execution Results:**

=== HTTP Server Verification: server === Test 1: Health endpoint... PASS (HTTP 200) Test 2: API response... PASS (valid JSON) Test 3: 404 handling... PASS (HTTP 404) === All tests passed ===


**Summary:** 3 passed, 0 failed

Example 3: Python Parser

User: /bulwark-verify src/parser.py

Output:
## Verification Script Generated

**Component:** src/parser.py
**Type:** File Parser
**Language:** Python

**Script location:** tmp/verification/parser_test.py

**To run manually:**

pytest tmp/verification/parser_test.py -v



Completion Checklist

Before completing bulwark-verify execution, verify ALL items:

Component Analysis (Step 3)

  • Target file read successfully
  • Component type identified (CLI, HTTP Server, File Parser, Database, Process Spawner, External API)
  • Project language detected (Node, Python, Rust, Generic)
  • assertion-patterns skill content loaded
  • component-patterns skill content loaded
  • bug-magnet-data context file loaded for component type
  • T0 + T1 edge cases loaded from bug-magnet-data

Script Generation (Step 4)

  • Sonnet sub-agent spawned (NOT generated by orchestrator)
  • Prompt includes component code, assertion patterns, component patterns
  • Prompt includes edge cases from bug-magnet-data (T0 + T1 values)
  • Script written to tmp/verification/{component_name}-verify.{ext}

Validation (Step 5)

  • Syntax validation command executed (node --check, python -m py_compile, bash -n)
  • Validation passed (or errors fixed and re-validated)

Edge Case Coverage

  • T0 boundary values included (empty string, zero, null)
  • T1 edge cases included if applicable (injection, unicode)
  • Destructive patterns excluded or marked as manual-only comments

Output

  • Summary presented to user with script location and run command
  • Log written to logs/bulwark-verify-*.yaml
  • Diagnostics written to logs/diagnostics/bulwark-verify-*.yaml
  • If --execute: Script executed, PASS/FAIL results reported

Do NOT return to user until all applicable checklist items are verified.


Related Skills

  • assertion-patterns (P2.1) - T1-T4 transformation patterns
  • component-patterns (P2.2) - Component-type verification templates
  • bug-magnet-data (P4.2) - Curated edge case test data