eval
Generate a complete Langfuse evaluation pipeline: auto-detect app type, create datasets, LLM-as-judge evaluators, scoring configs, and CI integration.
SKILL.md.agents/skills/eval/SKILL.md
GitHub/eval — Langfuse Evaluation Pipeline Generator
Generate evaluation setup for: $ARGUMENTS
- Discover —
agents/eval-architect.mdStep 1: Explore codebase, detect LLM app type (RAG, chatbot, agent, code-gen, summarization, classification, extraction), identify prompts, traces, and data flows - Design —
agents/eval-architect.mdSteps 2-3: Select 4-8 eval dimensions from the dimension matrix, generate seed dataset with 10-20 items covering happy path, edge cases, and adversarial inputs - Generate —
agents/eval-architect.mdSteps 4-5: Write completeevals/directory: config.yaml, evaluators (LLM judge + heuristic), runner.py, ci_eval.py, .env.example - Setup —
agents/eval-architect.mdStep 6: IfLANGFUSE_SECRET_KEYis set, run setup_langfuse.py. If not, skip gracefully with instructions. - Document —
agents/eval-architect.mdStep 7: Generate PM/SME-friendly README - Verify — Run verification checklist
Rules: Always include at least one heuristic evaluator. Never hardcode API keys. README must be understandable by non-developers.