extend-harness-module

Add parent directory to path when running eval scripts directly. Trigger: when executing eval/harness.py as main script.

SKILL.md

GitHub
<!-- L1:START -->

extend-harness-module

Add parent directory to path when running eval scripts directly.

Trigger: executing eval/harness.py as main script or importing from repoforge/cli.py

<!-- L1:END --> <!-- L2:START -->

Quick Reference

Task Pattern
Add parent to sys.path sys.path.insert(0, '..')
Run eval harness python -m eval.harness
Execute scenario python -m eval.harness run_scenario --scenario model --verbose

Critical Patterns (Summary)

  • Add parent to path: Insert '..' into sys.path before importing harness modules when running scripts directly
  • Use run_scenario: Call run_scenario() with scenario name to execute evaluation with optional LLM and verbose output<!-- L2:END -->
<!-- L3:START -->

Critical Patterns (Detailed)

Add parent to path

When running eval/harness.py directly, the module's parent directory must be added to sys.path to resolve sibling imports. This pattern uses pathlib.Path to dynamically resolve the project root and insert it into the path, ensuring all harness exports (make_fastapi_crud_module, score_trigger_precision, etc.) are accessible.

import sys
from pathlib import Path

# Add parent directory to path for direct script execution
project_root = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(project_root))

# Now imports from the module work
from eval.harness import make_fastapi_crud_module, run_scenario

Use run_scenario to execute evaluations

The run_scenario function orchestrates a single evaluation run with optional LLM integration. It accepts a scenario name, optional LLM instance, verbose flag, and facts dictionary. The function returns an EvalResult containing scores for trigger precision, code concreteness, pattern detection, and multilang coverage, enabling automated evaluation of LLM outputs against expected module patterns.

from eval.harness import run_scenario

result = run_scenario(
    scenario_name="model",
    verbose=True,
    facts=["extracted_fact_1", "extracted_fact_2"]
)
print(f"Trigger precision: {result.score_trigger_precision}")
print(f"Code concreteness: {result.score_code_concreteness}")

When to Use

  • Running eval/harness.py directly from any working directory
  • Executing scenario-based evaluations with python -m eval.harness
  • Importing harness functions from repoforge/cli.py or other modules
  • Debugging module score outputs (score_trigger_precision, score_code_concreteness)

Commands

# Run the harness with a specific scenario
python -m eval.harness run_scenario --scenario model --verbose

# Run all scenarios
python -m eval.harness run_all

# Execute from project root with parent path auto-resolved
python -m eval.harness --scenario auth --facts '["key=value"]'

Anti-Patterns

Don't: Skip sys.path modification when running directly

<Why it's wrong: Without adding the parent directory to sys.path, Python cannot resolve imports like from eval.harness import ..., raising `ModuleNotFoundError: No module named 'eval'. This breaks all harness functionality when running scripts outside the project package context.>

# BAD: Running without path setup
import sys
# Missing: sys.path.insert(0, '..')
from eval.harness import run_scenario  # ModuleNotFoundError
<!-- L3:END -->