<!-- generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten -->

Overview

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

Quick install

npm install -g nextroute   # or: npx nextroute
nextroute --version

Subcommands

eval

Example:

nextroute eval

eval suites

Example:

nextroute eval suites

eval list

Example:

nextroute eval list

eval get <suiteId>

Example:

nextroute eval get <suiteId>

eval create

Flags:

• --file <path>

Example:

nextroute eval create

eval run <suiteId>

Flags:

• -m, --model <id>
• --combo <name>
• --concurrency <n>
• --tag <tag>
• --watch

Example:

nextroute eval run <suiteId>

eval list

Flags:

• --suite <id>
• --status <s>
• --since <ts>
• --limit <n>

Example:

nextroute eval list

eval get <runId>

Example:

nextroute eval get <runId>

eval results <runId>

Flags:

• --failed

Example:

nextroute eval results <runId>

eval cancel <runId>

Flags:

• --yes

Example:

nextroute eval cancel <runId>

eval scorecard <runId>

Example:

nextroute eval scorecard <runId>

simulate [prompt]

Flags:

• --file <path>
• -m, --model <id>
• --combo <name>
• --reasoning-effort <level>
• --thinking-budget <n>
• --explain

Example:

nextroute simulate [prompt]

<!-- skill:custom-start --> <!-- Migrated from skills/nextroute-cli-eval/SKILL.md (preserved curated content) -->

NextRoute — CLI Evals

Requires the nextroute CLI. See CLI entry-point skill for install + global flags.

What are evals?

Evals are automated test suites that score LLM outputs against expected answers or rubrics. NextRoute stores suites and run results in its local database.

Eval suites

nextroute eval suites list                       # List all eval suites
nextroute eval suites list --json                # JSON output

nextroute eval suites get <suiteId>              # Full suite definition

Create a suite

nextroute eval suites create \
  --name "code-quality" \
  --rubric "exact-match" \
  --samples-file ./samples.jsonl                 # JSONL: {input, expected_output}

Rubric options: exact-match, contains, llm-judge, regex.

--samples-file format (one JSON object per line):

{"input": "What is 2+2?", "expected_output": "4"}
{"input": "Translate 'hello' to Spanish", "expected_output": "hola"}

Run an eval

nextroute eval suites run <suiteId> \
  --model claude-sonnet-4-6                      # Run suite against a specific model

nextroute eval suites run <suiteId> \
  --model gpt-4o \
  --watch                                        # Live TUI progress (EvalWatch)

The run is asynchronous. Use --watch for a live terminal dashboard or poll manually:

RUN_ID=$(nextroute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id')
nextroute eval get $RUN_ID

Manage runs

nextroute eval list                              # List all eval runs
nextroute eval list --json

nextroute eval get <runId>                       # Run details (status, model, score)
nextroute eval results <runId>                   # Per-sample results
nextroute eval scorecard <runId>                 # Full scorecard with pass/fail per sample
nextroute eval cancel <runId>                    # Cancel a running eval

Scorecard output

nextroute eval scorecard <runId> --output json

Response fields per sample:

{
  "id": "sample-1",
  "score": 0.95,
  "passed": true,
  "input": "What is 2+2?",
  "output": "4",
  "expected": "4"
}

Comparing models

Run the same suite against multiple models and compare:

for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do
  nextroute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}'
done

CI integration

# Run and fail CI if score drops below threshold
SCORE=$(nextroute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score')
python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)"

Errors

• suites create fails with invalid rubric → use one of: exact-match, contains, llm-judge, regex
• suites run returns model not found → verify model ID with nextroute models --search <name>
• eval get shows status: failed → check nextroute logs --search eval for error details
• scorecard returns empty results → the run may still be running; poll nextroute eval get <runId> until status is completed<!-- skill:custom-end -->