insights-report
Use when building a GEPA pipeline insights report for a completed or in-progress pipeline run, when asked to document pipeline results, or when comparing pipeline run metrics across experiments.
SKILL.mdclaude/insights-report/SKILL.md
GitHubGEPA Pipeline Insights Report
Build publication-quality insights reports for GEPA pipeline runs with data-driven diagrams and cross-run comparison tables.
Overview
An insights report documents a single pipeline run's performance: score trajectory, per-container analysis, feature assessment, and comparison against prior runs. Reports live in docs/insights_reports/ with diagrams in docs/images/.
When to Use
- User asks to "build an insights report" or "document pipeline results"
- A pipeline run has completed or is far enough along to analyze
- User wants to compare runs or assess new features
Environment Setup
Source project variables from .env before any data access:
set -a && source .env && set +a
Key variables used throughout this workflow:
| Variable | Used For | Example |
|---|---|---|
GOOGLE_CLOUD_PROJECT |
GCS client, Pipeline SDK | hybrid-vertex |
BUCKET_NAME |
GCS log/artifact bucket | my-pipeline-artifacts |
EXPERIMENT_NAME |
GCS path prefix | sup-chain-test-v1 |
AGENT_ENGINE_ID |
Memory Bank namespace | 5646775610664550400 |
GOOGLE_CLOUD_LOCATION |
Pipeline SDK region | us-central1 |
GOOGLE_CLOUD_MODEL_LOCATION |
Gemini endpoint | global |
Always use these variables — never hardcode project IDs, bucket names, or experiment names.
Quick Reference
| Step | Action | Tool/Skill |
|---|---|---|
| 1. Get pipeline metadata | Parameters, state, task details | inspect-vai-pipes skill |
| 2. Download GCS logs | Per-table JSONL + progress.json | gsutil or Python GCS client |
| 3. Extract metrics | Scores, phases, tools, data sources | Python script (inline) |
| 4. Read prior reports | Comparison metrics from existing reports | Read tool on docs/insights_reports/ |
| 5. Generate architecture diagram | GCP-styled 5-tier pipeline | gcp-diagram skill |
| 6. Generate data diagrams (4x) | Score trajectory, cross-run, container, feature | paperbanana skill |
| 7. Write report | Markdown with embedded images | Write tool |
Step 1: Pipeline Metadata
Use the inspect-vai-pipes skill with the run name:
/inspect-vai-pipes my-pipeline-run-YYYYMMDD-HHMMSS
This gives: state, parameters (epochs, concurrency, fitness_scaling_factor, novelty_weight, worker_timeout, explore_rate, topic), task details, and timing.
Run Name to GCS Path Mapping
Pipeline run names map to GCS paths by reformatting the timestamp:
my-pipeline-run-20260328-175751
↓
run-2026_03_28_17_57_51
Pattern: YYYYMMDD-HHMMSS → YYYY_MM_DD_HH_MM_SS
GCS base path:
gs://{bucket}/{experiment}/run-{YYYY_MM_DD_HH_MM_SS}/pipeline_root/evidence_reports/
Step 2: Download GCS Logs
from google.cloud import storage
import json
client = storage.Client(project='hybrid-vertex')
bucket = client.bucket('my-pipeline-artifacts')
prefix = '{experiment}/run-{timestamp}/pipeline_root/evidence_reports/'
# Discover tables
tables = set()
for blob in bucket.list_blobs(prefix=prefix):
parts = blob.name[len(prefix):].split('/')
if len(parts) >= 2 and parts[0] not in ('checkpoint_dir', 'code'):
tables.add(parts[0])
# Download per-table logs and progress
for table in sorted(tables):
log_blob = bucket.blob(f'{prefix}{table}/react_loop_fanout_log.json')
prog_blob = bucket.blob(f'{prefix}{table}/progress.json')
# Download and parse...
Step 3: Extract Metrics
Parse JSONL log entries. Each line has this structure:
{
"epoch": 5,
"evaluation": {
"overall_score": 0.85,
"scores": {"novelty": 0.7, "relevance": 0.9, "impact": 0.8, "actionability": 0.85, "specificity": 0.9, "trace_audit": 1.0},
"should_persist": true,
"justification": "..."
},
"evolution_phase": "Crossover",
"persona": "synthesis-crossover-1234",
"tools_used": ["generate_table_insights", "run_code_analysis"],
"tools_declared": ["generate_table_insights"],
"data_sources": ["1P", "GDELT", "Trends"],
"diversity_score": 0.95,
"source_dataset": "stores"
}
Key Metrics to Compute
| Metric | Formula |
|---|---|
| Scored | Count of entries with evaluation.overall_score |
| Persisted | Count where evaluation.should_persist == true |
| Floors | Count where overall_score <= 0.10 |
| Mean(NF) | Mean of scores > 0.10 |
| Trajectory | (late_third_mean - early_third_mean) / early_third_mean * 100 |
| Per-container | Group all above by source_dataset |
| Phase distribution | Group by evolution_phase |
Step 4: Read Prior Reports
Extract comparison metrics from these reports (read the Executive Summary tables):
| Report | File |
|---|---|
| Tuned 28-Ep (Cold) | docs/insights_reports/tuned_gepa_50epoch_report.md |
| 25-Ep Asteroid | docs/insights_reports/25epoch_evolution_report_20260325.md |
| 30-Ep Supply Chain | docs/insights_reports/supply_chain_30epoch_report_20260326.md |
| 25-Ep H5 Cal | docs/insights_reports/supply_chain_25epoch_h5_calibration_20260327.md |
| 25-Ep H5v2 | docs/insights_reports/supply_chain_25epoch_h5v2_20260328.md |
| 50-Ep v1 (WIP) | docs/insights_reports/supply_chain_50epoch_v1_20260330.md |
Step 5: Generate Architecture Diagram
Use the gcp-diagram skill. Write a text description to /tmp/pipeline_report/diagram_architecture.txt, then invoke:
/gcp-diagram /tmp/pipeline_report/diagram_architecture.txt "GEPA {N}-Epoch Pipeline Architecture"
The description should cover the 5-tier GEPA flow: Discovery → Synthesis → Execution → Evaluation → Persistence. Annotate any new features being tested in this run.
Save output to docs/images/{prefix}_architecture.png. Run the icon overlay script after generation.
Step 6: Generate Data Diagrams (PaperBanana)
For each data diagram, write a text description file to /tmp/ then use the paperbanana skill.
Required Diagrams (5)
1. Score Trajectory — Multi-line chart: per-container + aggregate (dashed) mean(NF) over epochs, with early/mid/late shaded bands and 3-epoch smoothing
mcp__paperbanana__generate_diagram(source_context=<contents of /tmp/{prefix}_score_trajectory.txt>, caption="Score trajectory over {N} epochs")
2. Cross-Run Comparison — Dual-panel grouped bar chart: left panel (Mean(NF) + Trajectory%), right panel (Persist% + Floor%) across all runs
mcp__paperbanana__generate_diagram(source_context=<contents of /tmp/{prefix}_cross_run.txt>, caption="Cross-run performance comparison")
3. Container Performance — Horizontal grouped bars (scored/persisted/floors) per table, sorted by Mean(NF), with trajectory annotations
mcp__paperbanana__generate_diagram(source_context=<contents of /tmp/{prefix}_containers.txt>, caption="Per-container performance breakdown")
4. Container Trajectories — Grouped bars showing Early/Mid/Late mean scores per container + aggregate, with trajectory % annotated above each group
mcp__paperbanana__generate_diagram(source_context=<contents of /tmp/{prefix}_container_trajectories.txt>, caption="Per-container learning curves")
5. Feature Assessment (if new features tested) — Status cards for new features
mcp__paperbanana__generate_diagram(source_context=<contents of /tmp/{prefix}_features.txt>, caption="New feature assessment dashboard")
Diagram Text File Format
Each text file should contain:
- Title
- Chart type (line chart, bar chart, status cards, etc.)
- Actual data values (not placeholders)
- Axis labels and annotations
- Color scheme hints
- Layout preferences
Save outputs to docs/images/{prefix}_*.png.
Step 7: Write the Report
See references/report-template.md for the full section structure. The report follows this skeleton:
- Header — Title, pipeline metadata block, key result one-liner
- Executive Summary — N-run comparison table, 3-5 key findings
- Methodology — Config changes table, architecture diagram, parameter tuning rationale (if config differs from prior run)
- Score Trajectory — Early/mid/late epoch-range table, trajectory chart, per-epoch mean(NF) trend analysis
- Score Distribution — Floor analysis (critic failures vs legitimate lows), per-dimension scores (novelty, relevance, impact, actionability, specificity, trace_audit)
- Container Analysis — Per-container performance table, per-container trajectory (early/mid/late with pattern labels), per-container trajectory chart, phase distribution with quality analysis, tool usage, data source distribution
- Adaptive Threshold Analysis (if FITNESS_SCALING_FACTOR changed) — Threshold trajectory over epochs, headroom gap (mean score minus threshold), death spiral risk assessment
- Concurrency Analysis (if concurrency issues observed) — Three bottleneck model (Memory Bank writes, reads, Gemini API), effective throughput formula, duration analysis, recommended config by run length
- Feature Assessment (if new features tested) — Per-feature: status (Active/Disabled/Partial), mechanism, quantitative evidence, verdict (Keep/Modify/Remove)
- Cross-Run Comparison — Full comparison table, controlled pairwise delta analysis against closest control run (same topic, fewest param differences)
- Discussion — Why trajectory is positive/negative/flat, cold vs warm start effects, which phases drove quality, concurrency collapse patterns, novelty saturation (if novelty < 0.40), late-epoch quality dips
- Recommendations — Next run config table (current vs recommended vs rationale), immediate actions, future work
- Appendix — Per-epoch detail table, pipeline parameters dump, GCS artifact paths, diagram list with generation method
File Naming Convention
docs/insights_reports/{topic}_{N}epoch_{variant}_{YYYYMMDD}.md
Examples:
supply_chain_25epoch_h5v2_20260328.mdsupply_chain_50epoch_v1_20260330.md
Image Reference Format

Common Mistakes
| Mistake | Fix |
|---|---|
| Using raw Gemini API for diagrams | Use the paperbanana and gcp-diagram skills |
Computing epoch as index / 10 |
Use the actual epoch field from log entries |
| Missing per-container trajectory | Always include early/mid/late breakdown per table |
| Comparing without controlling for config | Note which params differ in comparison tables |
| Not mentioning Memory Bank state | Always state cold vs warm start |
| Hardcoding GCS paths | Derive from experiment + run name |
| Using Cloud Logging scores | GCS JSONL logs are authoritative (structured, complete) |
| Skipping the icon overlay | GCP diagrams need official icons via overlay_icons.py |