ai-model-route-evaluator
Compare current AI models, providers, gateways, subscriptions, local routes, and free/paid endpoints for a concrete task. Evaluate exact model identity, modality, Turkish and multilingual quality, cod
SKILL.mdskills/common/ai-model-route-evaluator/SKILL.md
GitHubAI Model Route Evaluator
Choose a route for the user's task, not a universal leaderboard winner. Model catalogs, aliases, limits, prices, and product entitlements are time-sensitive; verify them from current primary sources.
Workflow
- Define the job. Resolve the actual task, input/output modalities, languages, context size, tool/agent needs, latency tolerance, budget, data sensitivity, local hardware, and whether a free route is a hard constraint. Ask only for a missing fact that could change the winner.
- Resolve exact identities. Record provider, product/plan or gateway, exact model ID/version/date, endpoint, region when relevant, and whether a name is a floating alias. Never treat a gateway alias as a stable model identity.
- Verify capability and access. Use official catalogs, model cards, pricing, rate-limit/usage documentation, privacy/data-use terms, and product help. Separate API access from consumer subscription access.
- Select task-fit evidence. Prefer current, version-pinned evaluations that match the required language, modality, agent/tool use, and difficulty. Do not infer Turkish or multimodal quality from an English text benchmark alone.
- Measure locally when practical. Run the same representative prompt/input, record settings, retries, time, tokens/cost, success criteria, and failures. Do not compare outputs produced under materially different scaffolds as if only the model changed.
- Assess privacy and operational risk. Never route API keys, private code, customer data, health/legal/financial records, or personal data through an untrusted or vaguely governed free endpoint. State retention/training facts only when verified for that exact product and plan.
- Score by scenario. Give one winner per meaningful task class, a fallback, and explicit disqualifiers. Use weighted scores only after showing the raw facts; do not hide a hard requirement inside an average.
- Conclude conditionally. State what to use now, for which task, under which plan/route, why, when to switch, and what remains uncertain.
Read references/evaluation-scorecard.md for the comparison schema.
Output contract
## Decision
<task-specific winner, exact route, and one-line reason>
| Route | Exact model/version | Best for | Limits/cost | Privacy | Evidence confidence |
|---|---|---|---|---|---|
## By task
- <task>: <winner> — <reason and disqualifier>
## What to do
<setup or selection steps, fallback, and recheck trigger>
## Uncertainty
<floating aliases, missing Turkish/multimodal evidence, unmeasured latency, or plan ambiguity>
Avoid provider fandom, copied leaderboard rank, guaranteed quality, or a single winner across unrelated tasks.