issue-debugging

Systematic methodology for issue debugging including root cause analysis, impact mapping, tiered validation plans, and confidence assessment. Use when analyzing bugs, fixing issues, or validating fixe

SKILL.md

GitHub

Issue Debugging

Systematic methodology for debugging that prevents the "first error fix and declare done" anti-pattern. Ensures root cause analysis, impact assessment, and verified fixes.


When to Use This Skill

Load this skill when the user request matches ANY of these patterns:

Trigger Pattern Example User Request
Bug investigation "Why is X failing?", "Debug this error", "Investigate this issue"
Test failures "Tests are failing", "Fix the broken tests", "Why did CI fail?"
Error analysis "Getting this error...", "Exception in...", "Stack trace shows..."
Regression hunting "This used to work", "Something broke after...", "Regression in..."
Fix validation "Did the fix work?", "Verify this change", "Make sure it's fixed"
Root cause requests "What's causing this?", "Find the root cause", "Why does this happen?"

DO NOT use for:

  • New feature implementation (no existing bug)
  • Refactoring without reported issues
  • General code questions

Overview

The Problem

AI (and developers) often fix the first visible error and declare victory without verifying end-user functionality works. This leads to:

  • Symptom fixes instead of root cause fixes
  • Regressions in seemingly unrelated areas
  • False confidence in fix completeness

The Solution

This skill provides:

  1. Root Cause Analysis - Find the actual cause, not just symptoms
  2. Impact Analysis - Map affected areas and dependencies
  3. Validation Planning - Tiered test priorities (P1/P2/P3)
  4. Confidence Assessment - Objective fix reliability rating
  5. Escalation Triggers - When manual testing is required

Root Cause Analysis

The 5 Whys Method

Iteratively ask "Why?" to drill past symptoms to root cause:

Problem: Login fails with 500 error
├─ Why? → API returns null for user profile
├─ Why? → Database query found no records
├─ Why? → Migration didn't run on deploy
├─ Why? → Deployment script skipped migration step
└─ Why? → Environment variable missing

Root Cause: Missing environment variable (not the 500 error)

Key Principle: Focus on HOW and WHY the defect occurred, not just WHERE.

Hypothesis-Driven Debugging

Apply scientific method:

Step Action
Observe Gather data: errors, logs, stack traces, reproduction steps
Hypothesize Form a falsifiable hypothesis about the cause
Experiment Design tests to prove/disprove hypothesis
Conclude If confirmed, proceed. If rejected, form new hypothesis
Document Write down hypotheses tested (prevents circular investigation)

Binary Search / Bisection

For isolating bugs in time or space:

  1. Place checkpoint at midpoint
  2. Determine if bug is before or after
  3. Repeat on remaining half
  4. Continue until exact location found

Efficiency: O(log n) - approximately 7 tests for 100 commits.


Impact / Dependency Analysis

Mapping Procedure

For each affected file, identify:

Affected Code: src/auth/login.ts

Upstream (What calls this?):
├─ src/api/auth-routes.ts → POST /login
├─ src/middleware/session.ts → validateSession()
└─ src/pages/login.tsx → handleSubmit()

Downstream (What does this affect?):
├─ User dashboard → fetches profile on load
├─ Settings page → assumes profile exists
└─ API responses → include user context

Risk Scope Assessment

Scope Definition Validation Approach
Isolated Self-contained, no affected areas Unit tests only
Medium 1-5 affected areas Integration tests
Broad >5 affected areas, cross-cutting Integration + E2E + manual

Debug Report Format

Location

Debug reports are written to: logs/debug-reports/{issue-id}-{timestamp}.yaml

This location is separate from standard agent logs to:

  • Enable easy reference by downstream pipeline stages
  • Keep debug context grouped by issue
  • Allow FixValidator to read validation plans

Schema Overview

debug_report:
  metadata:
    issue_id: "{id}"
    timestamp: "{ISO-8601}"
    analyzer: "bulwark-issue-analyzer"
    complexity: low | medium | high

  analysis:
    symptom: "{user-visible problem}"
    root_cause: "{underlying reason}"
    fix_approach: "{recommended approach}"

  impact_analysis:
    affected_files: [...]
    upstream_dependencies: [...]
    downstream_effects: [...]
    risk_scope: isolated | medium | broad

  validation_plan:
    tests_to_execute:
      - path: "{test file}"
        reason: "{why this test}"
        priority: 1  # P1=must, P2=should, P3=nice-to-have
    functionalities_to_validate: [...]

  confidence_criteria:
    high: [...]
    medium: [...]
    low: [...]

  debug_journey:  # Required for medium/high complexity
    hypotheses_tested: [...]

Full schema: See references/debug-report-schema.md


Validation Plan

Tiered Test Priorities

Priority Definition Action
P1 (Must) Direct tests of affected function Always run
P2 (Should) Integration tests for upstream/downstream Run if time/tokens allow
P3 (Nice-to-have) E2E tests, edge cases Run if available

Selection Criteria

P1 Tests (always run):

  • Tests in the same file/module as the fix
  • Tests named after the affected function
  • Tests that were failing before the fix

P2 Tests (run if possible):

  • Tests for upstream callers
  • Tests for downstream consumers
  • Tests for related functionality

P3 Tests (run if available):

  • E2E tests that include the affected flow
  • Performance tests
  • Edge case tests

Functionality Validation

Beyond tests, list user-level verifications:

  • "User can log in successfully"
  • "Dashboard displays correct data after login"
  • "Session persists across page refresh"

Confidence Rubric

Assessment Criteria

Level Criteria
High All P1-P2 tests pass, no regression, at least one functionality verified
Medium P1 tests pass, some P2-P3 skipped or not applicable
Low Tests cannot reliably validate, broad risk scope with untested paths

Determining Complexity

Complexity Indicators
Low Single file, isolated change, clear cause
Medium 2-5 files, some dependencies, requires investigation
High >5 files, cross-cutting, unclear cause, multiple hypotheses needed

Escalation Triggers

Manual testing is required when ANY of:

  • Confidence level is low
  • Risk scope is broad AND confidence is not high
  • Any functionality cannot be validated via automated tests
  • Security-related fix without security test coverage

Escalation Message Format

When escalation is triggered, the Orchestrator must inform the user:

Manual testing required for:
- [functionality 1]
- [functionality 2]

Reason: [why automated validation insufficient]

Recommended manual test steps:
1. [step]
2. [step]

Debug Journey Logging

When Required

Complexity Debug Journey Required?
Low Optional
Medium Required
High Required

Format

debug_journey:
  started_at: "{timestamp}"
  completed_at: "{timestamp}"

  hypotheses_tested:
    - hypothesis: "Database connection timeout causing failure"
      tested_at: "{timestamp}"
      method: "Checked DB logs and connection pool stats"
      result: rejected
      evidence: "DB logs show successful queries, pool not exhausted"

    - hypothesis: "Null profile object when user has no data"
      tested_at: "{timestamp}"
      method: "Added logging at profile access point"
      result: confirmed
      evidence: "Stack trace shows NPE at profile.name access"

  investigation_path:
    - "Started with error log analysis"
    - "Ruled out infrastructure issues"
    - "Narrowed to application code"
    - "Identified profile access as failure point"

Anti-Patterns

What NOT to Do

Anti-Pattern Description Correct Approach
Shotgun Debugging Random changes hoping bug disappears Hypothesis-driven approach
Fix Without Verify Declaring "fixed" without running tests Always verify at code + user level
Symptom Fixing Adding workarounds instead of root cause fix Use 5 Whys
Blind AI Trust Accepting suggestions without verification Test the failing scenario
Full Regression Running 4000+ tests after every fix Use tiered validation
Circular Investigation Repeating tested hypotheses Document debug journey

Full checklist: See references/anti-patterns.md


Quick Reference

Before Fixing

  • Symptom identified and documented
  • Root cause analysis completed (not just symptom)
  • Impact analysis completed (upstream/downstream)
  • Risk scope assessed (isolated/medium/broad)
  • Complexity determined (low/medium/high)

After Fixing

  • Fix implemented
  • P1 tests pass
  • P2 tests pass (or documented why skipped)
  • Confidence level assessed
  • Manual testing escalated if required
  • Debug journey documented (if medium/high complexity)
  • Debug report written to logs/debug-reports/

Complexity → Actions

Complexity Debug Journey Validation Escalation
Low Optional P1 tests Only if low confidence
Medium Required P1 + P2 tests If broad scope
High Required P1 + P2 + P3 Almost always

Integration

Pipeline Integration

This skill is used by the Fix Validation pipeline:

IssueAnalyzer (loads this skill, produces debug report)
|> FixWriter (reads debug report for context)
|> TestWriter (reads validation plan)
|> FixValidator (executes validation plan, assesses confidence)
|> CodeReviewer (reviews all, makes approval decision)

Agent Integration

Agent Usage
bulwark-issue-analyzer (P1.2) Loads via skills: issue-debugging, produces debug report
bulwark-fix-validator (P1.3) Loads via skills: issue-debugging, executes validation plan

Artifact Flow

IssueAnalyzer
    └─> logs/debug-reports/{issue-id}.yaml
              │
              ├─> FixWriter (reads root cause, fix approach)
              ├─> TestWriter (reads validation plan)
              ├─> FixValidator (executes tests, assesses confidence)
              └─> CodeReviewer (reviews everything)

Related Skills

  • subagent-prompting (P0.1) - 4-part template for agent invocation
  • subagent-output-templating (P0.2) - Output format for logs
  • pipeline-templates (P0.3) - Fix Validation pipeline definition