Synthesize Case Study and Improvement Plan

Read all observation data captured by the case study harness, analyze it, and produce two Markdown deliverables. Follow these steps in order.

Step 1: Locate the harness engineering guide

The improvement plan must reference actual guide chapters, sections, and templates by name. To do this accurately you need access to the guide's content. Ask the user:

To write an accurate improvement plan, I need to reference the actual chapters and sections in the harness engineering guide. Do you have the guide repository cloned locally? If so, provide the path (e.g., ~/source/harness-engineering-guide). Otherwise, I'll fetch what I need from the public GitHub repository.

Then:

• If the user provides a local path: Use Read to read README.md and files under guide/ from that path. This is the preferred option -- it is faster, works offline, and always reflects the user's current version.
• If the user says no or wants the remote option: Use WebFetch to fetch from https://github.com/brandonavant/harness-engineering-guide. Start with the README for the chapter listing, then fetch specific chapters under guide/ as needed.

At minimum, read the guide's README.md (which contains the full chapter listing and structure overview). Then read specific chapters that are relevant to the observations found in the data -- you do not need to read all 11 chapters, only those you will reference in the improvement plan.

Keep the guide content available in context for Steps 5 and 6.

Step 2: Run the reader script

Invoke the script to read, validate, merge, and summarize all observation data:

python3 "${CLAUDE_SKILL_DIR}/scripts/read_observations.py"

The script reads all seven JSONL log files, merges entries chronologically, computes summary statistics, and outputs a single JSON object to stdout with two keys:

• "summary" -- statistics (entry counts by type, session count, date range, friction tool breakdown, harness change file breakdown, manual observation category breakdown, tool use breakdown)
• "entries" -- all entries merged and sorted by timestamp

If the script reports errors or missing files, note which event types have no data but continue with whatever data is available.

Step 3: Analyze the data

Using the script's output, work through these analytical lenses:

• User intent trail: What did the user ask for at each turn? How did requests evolve over the course of the build? Were there corrections or pivots? Use user_prompt entries to reconstruct the narrative arc of what the user was trying to accomplish.
• Tool usage patterns: Which tools were used most frequently? What files were touched most often? Look for Read-then-Edit chains that indicate exploration followed by action. Use tool_use entries to understand the agent's working patterns.
• Harness evolution: How did harness files (.claude/, CLAUDE.md) change over time? What was added, modified, or removed? Were changes concentrated early or spread throughout?
• Friction patterns: What types of failures occurred? Were they clustered around specific tools, tasks, or time periods? Were any recurring?
• Session rhythm: How many sessions occurred? How long were they (based on turn counts between session boundaries)? Were there sessions with unusually high friction?
• Manual observations: What did the user flag as noteworthy? Group by category.
• Successful patterns vs. friction: Which harness decisions produced good outcomes? Which produced friction or required human intervention?

Step 4: Create the output directory

mkdir -p case-study-harness/output

Step 5: Write the case study draft

Write the case study to case-study-harness/output/case-study-draft.md.

Title

Use this format:

# Case Study: [Project Name] -- Building with Harness Engineering

Use the project name from the repository if identifiable; otherwise use a generic title.

Required sections

1. Executive Summary -- 2-3 paragraph overview of the project, the harness approach used, and the headline results.

2. Project Context -- What was built. What harness mechanisms were used (CLAUDE.md, rules, skills, hooks). How many sessions the build spanned.

3. Harness Evolution -- How the harness changed over the course of the build. Cite specific harness_change entries by timestamp. Note what was added early vs. what emerged later.

4. What Worked -- Successful patterns and effective harness decisions. Cite manual entries with category successful_pattern and any harness_change entries that correlate with reduced friction. Use user_prompt entries to provide context for why a decision was made. Each claim must reference at least one log entry.

5. What Failed and How It Was Addressed -- Friction events, human overrides, and how they were resolved. Cite friction entries and manual entries with category human_override or friction. Use user_prompt and tool_use entries to reconstruct the sequence of events leading to and resolving the failure. Describe the failure, the resolution, and whether a harness change resulted.

6. Context Architecture Observations -- How context was distributed across CLAUDE.md, rules, skills, and hooks. Cite manual entries with category context_architecture and relevant harness_change entries. Note any placement decisions that worked well or poorly.

7. Lessons Learned -- Key takeaways distilled from the evidence above. Each lesson must cite at least one supporting log entry.

Citation format

Cite log entries inline using this format: [event_type @ timestamp]

Examples:

• [friction @ 2026-04-01T14:30:22+00:00]
• [manual/successful_pattern @ 2026-04-02T09:15:00+00:00]
• [harness_change @ 2026-04-01T10:00:00+00:00]

For manual observations, include the category after a slash as shown above.

Step 6: Write the improvement plan

Write the improvement plan to case-study-harness/output/improvement-plan.md.

Title

# Harness Guide Improvement Plan

Required sections

1. Summary -- Brief overview of proposed changes and their motivation.

2. Recommendations -- A numbered list of specific, actionable changes to the harness engineering guide. Each recommendation must include:

   • What to change -- the specific guide chapter, template, or section affected. Use the actual chapter titles and section headings from the guide content you read in Step 1. Do NOT guess or invent chapter names.
   • Why -- the evidence from this build that motivates the change
   • Evidence -- one or more log entry citations using the [event_type @ timestamp] format
   • Priority -- high, medium, or low based on how frequently the issue occurred or how severe its impact was

3. Patterns to Document -- Successful patterns discovered during the build that the guide does not currently cover. Cite the supporting evidence.

4. Antipatterns to Warn About -- Mistakes or friction patterns that the guide should warn against. Cite the supporting evidence.

Step 7: Confirm

Report to the user:

• How many total log entries were processed
• Entry count per event type
• How many recommendations the improvement plan contains
• The paths to both output files