choosing-models
Picks models for AQVEN agents on any provider from its data and probes: provider_options, rate limits, fallbacks. Use first when an agent costs too much or needs a cheaper or fallback model, before ed
SKILL.mdpackages/aqven/src/aqven/agent_plugin/skills/choosing-models/SKILL.md
GitHubMUST
- The owner picks providers and models, and any judges he named ("Owner's rules" in
AGENTS.md). Change that set, or one setting across many agents, only after the owner says yes. - When a check contradicts what runs showed, report it to the owner as a likely engine bug, with numbers (runs that passed, the code the check gave). Never swap models to satisfy the check.
- An agent file has no key for what a model can do: the provider's model data suggests, the real node proves. A
new model gets
models check --live, then one call on the real node with its output type and the input kind it will read (image, audio, video, PDF, long text).--livesends one small text request per output mode: it proves the mode, not the media, the length or the schema. - Set reasoning on purpose in every agent file, with the key its provider reads; never leave the default
unrecorded. The level is a factor like the model: start where the owner's latency and cost contract points, and
measure it (an
agentfactor over agent files that differ only in reasoning) before calling it right. - Facts in an agent's
descriptionname their source: a probe, a series id, a provider's model page, or "from another project, unverified here". - A 429 is not fixed with
limits.rpm: the model's rate-limit lane and the provider'son_rate_limithandle it.
Procedure
| # | Step | Exit criterion |
|---|---|---|
| 1 | Read "Owner's rules": the providers allowed, price ceiling, what is banned, any judges he named; providers in aqven.yaml shows what is declared |
the frame is known |
| 2 | Candidates from each allowed provider's own model data, not from names or memory: a direct provider's model pages and docs (its models endpoint needs the owner's key: ask him to run it); an aggregator's public catalogue (step 6); what a local or self-hosted server serves. Across model families where the allowed providers offer them. Many candidates: screen, then confirm (designing-experiments) |
a table of enough candidates to span the price and capability range the frame allows, from several families where available |
| 3 | Per candidate, whatever the provider, one column each with its source: the input kinds the flow sends (text, image, PDF, audio, video); structured output for the node's type; tool calling when the agent has tools, mcp_servers or subagents; context window against the longest input plus the answer; the longest answer; price per input and output token, and reasoning, image or audio prices; reasoning controls (off, efforts, mandatory); the rate limits of the owner's key; retention against the provider's data_policy |
no empty cell; what no source states is written as unknown |
| 4 | Name the model <provider id>:<model name>: the id is an entry of providers in aqven.yaml (E_PROVIDER_UNKNOWN otherwise), the name is the provider's own spelling. Direct: anthropic:<model id>, google:<model id>, openai:<model id>; bedrock:<model or inference profile id>; aggregator: openrouter:<author>/<slug>; local: ollama:<tag> with base_url on the provider; any other OpenAI-compatible server: a provider with kind: "openai_compatible", base_url and an id outside the catalog. anthropic, google, groq, mistral, cohere, bedrock, huggingface and xai need their extra, uv add "aqven[<name>]" (E_PROVIDER_EXTRA_MISSING) |
aqven_check clean on every model string |
| 5 | The agent file: description with the source of every claim; settings.max_tokens above the expected output plus any reasoning; settings.provider_options for what only this provider takes, reasoning first, at a starting level taken from the latency and cost contract, in the values its model pages list: reasoning_effort on OpenAI Chat Completions (openai-chat), reasoning with effort on OpenAI Responses (openai), thinking on anthropic, reasoning on openrouter, thinking_config on google, reasoning_effort on xai and on mistral (none or high), the model's own field on bedrock. google, mistral and xai send only the keys the provider catalog lists, cohere none: W_PROVIDER_OPTIONS_IGNORED names every key left out, and a model whose reasoning no key reaches keeps its default, so write that in description and measure latency. output.mode from models check; output.on_error, output.on_refusal, output.on_truncated |
aqven_check clean; reasoning set, or its default recorded |
| 6 | The provider is an aggregator (openrouter and similar): run "When the provider is an aggregator" below for every candidate on it |
upstreams chosen, single-upstream candidates flagged |
| 7 | The provider in aqven.yaml: on_rate_limit auto (default: the model pauses for retry-after, else 2 s doubling to 60 s, parallel calls halve and grow back, a call gives up after 10 attempts or 120 s), fixed with retry_wait_seconds and retry_attempts, or fail to hand over to fallback_models at once; limits.concurrency is the starting parallelism per model (8 when unset), limits.rpm only the provider's own limit. Every <provider>:<model> is a lane of its own |
the strategy fits the latency contract |
| 8 | fallback_models that read the same input kinds, inside "Owner's rules", on the same provider or another declared one. settings.provider_options belong to the agent: every fallback gets the same keys, and a provider may reject a key it does not know or leave it out (W_PROVIDER_OPTIONS_IGNORED), so a fallback on another provider needs keys both take, proven by its own probe. E_OUTPUT_MODE_UNSUPPORTED checks the output modes of every model of the agent |
the fallback stays inside the frame and accepts the agent's options |
| 9 | Does the model take the input kind and its size (image, PDF, audio, video, a long document against the context window): the engine keeps no table. Read the provider's data, then one run on a real case of that kind and size: run_start {flow_id, mode: "live", dataset_item_id, agent_overrides: {<node_id>: <candidate agent>}} puts the candidate on the real node with no flow edit and no series; a provider refusal comes back as MODEL_FEATURE_UNSUPPORTED naming the attachment |
every input kind proven by a run |
| 10 | Prove it on the real node: uv run aqven models check <agent> --project <package> --live --provider-options '<the agent's provider_options as JSON>' (the modes); uv run aqven models shapes <agent> --project <package> --live for a nested output; then run_start with agent_overrides on a real case of each input kind, as at step 9. One run proves one case; the model decision is an agent factor (step 11). Read latency, tokens_out and error codes; a model three times slower than the rest goes to the owner before he finds it |
every agent ok, the real-node run finished, latency within the contract, outliers reported |
| 11 | Compare models on the project's labelled data with a negative control, as an agent factor (designing-experiments); synthetic data is only a sanity check |
the model decision rests on real labels |
| 12 | One change at a time. A setting rolled out to many models (reasoning, max_tokens, routing) waits for the owner's yes, then each model is measured before and after on the same few cases: run_start with agent_overrides per case, or a small dev series read with series_outputs (latency_ms, cost_usd per row); one case of latency is noise |
a before-and-after table per model |
The CLI probes read keys from the environment and <package>/.env. Keys saved in Studio settings are visible only
to the project server: when your shell has no key, do not search for it; ask the owner to run the probe.
A direct provider, reasoning switched off through the key Anthropic reads (the example shows the key; the level is the node's own, measured):
description: "Sorts support tickets into queues; prompted mode and PDF input proven by run <run_id>"
model: "anthropic:claude-haiku-4-5"
settings:
max_tokens: 4000
provider_options:
thinking:
type: "disabled"
output:
mode: "prompted"
on_refusal: "retry"
On OpenAI Chat Completions the same agent takes reasoning_effort under provider_options instead, with one of
the efforts its model page lists.
When the provider is an aggregator
An aggregator (OpenRouter, a gateway) serves one model through several upstream providers, and they differ in parameters, quantization, limits and uptime. For each candidate on it:
| # | Sub-step | Exit criterion |
|---|---|---|
| a | The catalogue. OpenRouter: curl -s 'https://openrouter.ai/api/v1/models?input_modalities=<kind>&supported_parameters=structured_outputs', fields architecture.input_modalities (text, image, file, audio, video), context_length, pricing, supported_parameters (structured_outputs, response_format, tools, reasoning), reasoning (mandatory, supported_efforts, default_enabled) |
the step 3 columns filled from it |
| b | The upstreams: curl -s https://openrouter.ai/api/v1/models/<author>/<slug>/endpoints, fields tag (the slug provider.order takes), provider_name, quantization, max_completion_tokens, supported_parameters, supports_tool_choice, status, uptime_last_30m. A candidate with one upstream has nowhere to send a 429: flag it |
single-upstream candidates flagged |
| c | Routing in settings.provider_options.provider: order, allow_fallbacks, require_parameters: true, quantizations. A pinned order with allow_fallbacks: false leaves an upstream 429 nowhere to go: allow fallbacks or set fallback_models. routing on the provider in aqven.yaml replaces the agent's provider object: put data_collection and zdr into the agent's instead |
an upstream order chosen |
| d | reasoning per model at the level chosen at step 5 (effort, down to "none" or "minimal", or enabled: false); where the catalogue's reasoning.mandatory is true, the lowest of supported_efforts is the floor |
reasoning set per model |
description: "Sorts support tickets into queues; prompted mode and PDF input proven by run <run_id>"
model: "openrouter:google/gemini-2.5-flash-lite"
settings:
max_tokens: 4000
provider_options:
provider:
order:
- "google-vertex"
- "google-ai-studio"
allow_fallbacks: true
require_parameters: true
reasoning:
enabled: false
output:
mode: "prompted"
on_refusal: "retry"
Pitfalls
| What goes wrong | Do instead |
|---|---|
| Candidates taken from one provider's list while "Owner's rules" allowed several | step 2 over every allowed provider |
| A weaker model picked because its profile was unknown | the provider's data suggests, the real node decides |
Reasoning and output.mode: tool set without a live probe: every attempt MODEL_FEATURE_UNSUPPORTED |
models check --live first |
The profile said tool works, live it gave MODEL_NO_STRUCTURED_OUTPUT; a big series ran without --live |
probe live before a series |
Every model passed models check --live, then the real node failed in four ways: the schema too large, no upstream for the request's parameters, tool_choice unsupported, no structured output |
one call on the real node, with its output type and input kind |
One variant failed every attempt with MODEL_FEATURE_UNSUPPORTED and nobody looked until the series ended |
series_get view: "summary" at the first snapshot: failure groups per variant (running-series) |
A candidate re-checked with agent_overrides on three cases and reported as better than the current model |
one run proves one case; a comparison is an agent factor on labelled data (step 11) |
| Reasoning left at each model's default: two models took 80 s per call against 5 s for the rest, and the owner noticed first | decide reasoning when the agent is written; report latency outliers from the first run |
| A "low" reasoning effort switched reasoning on for a model that had it off, and latency and price jumped | set reasoning explicitly per model and measure |
| A reasoning key written for one provider, copied to an agent on another: rejected, or dropped without a word | the key that provider documents, tried with --provider-options first |
provider_options in another provider's keys on a google, mistral or xai model, believed to work |
aqven_check without W_PROVIDER_OPTIONS_IGNORED: the keys the provider catalog lists for that provider |
A fallback on another provider failed every call on the agent's provider_options |
fallbacks whose providers take the same keys, each probed |
Truncation at max_tokens misread as a model or prompt failure |
finish_reason: length, truncated: on a model without reasoning raise settings.max_tokens; on a reasoning model the hidden reasoning used the budget (tokens_out near the limit, little visible answer), so cap reasoning or switch it off |
| A block of settings rolled out to every model at once and judged on one case's latency | only when the owner asks; before and after per model on several cases |
A claim from another project written into an agent's description as a fact |
cite a probe, a series or a model page, or mark it unverified |
OUTPUT_SCHEMA_REJECTED on a large enum list, answered with a model swap |
a contract problem: designing-output-contracts |
| Frontier models picked without looking at price, or left as fallbacks after the owner banned them | "Owner's rules" bound fallbacks too |
| Two models tied on synthetic cases; on real labelled cases they split, and one also flagged a refund request on tickets that only asked for an invoice copy | compare on real labelled data with a negative control |
| A check rejected a model that real runs had shown working, and the owner's model set was changed without asking | report a likely engine bug with the numbers, keep the owner's set |
| On an aggregator, a candidate served by one upstream answered 429 to every probe | flag it at sub-step b; fallback_models |
On an aggregator, allow_fallbacks: false on an agent pinned to two upstreams, inside a parallel whose join needs every branch: upstream 429s turned a holdout series invalid |
allow fallbacks or add fallback_models |
rpm lowered to fight 429 from a paid model |
the lane handles it; rpm is the provider's own limit only |
| The key searched for in the shell environment | the server holds Studio keys; ask the owner |
On an aggregator, quantization unknown on closed models read as missing data |
it is normal for closed models |
Tools and commands
uv run aqven models check <agent> --project <package> --live, with--provider-options '<json>'to try a provider setting before writing it; the probe does not read the agent's ownprovider_options.uv run aqven models shapes <agent> --project <package> --live.- A provider's own model pages; an aggregator's public catalogue with
curl, as above. AQVEN has no catalogue command. - macOS has no GNU
timeout: run a long probe in the background instead of wrapping it. - Agent files for many models come from a builder in
scripts/(building-flows). aqvenMCPaqven_check,run_start(mode: "live",dataset_item_id,agent_overrides),run_get_node,series_get(view: "summary"),series_outputs.uv run aqven run <flow_id> --input <file.json> --agent <node>=<agent>goes through the project server when one runs on this root, and starts an engine of its own only when none does; it takes a JSON input, not a dataset case.
References
references/integrations/model-providers.md: declaring a provider, the model name per kind of provider, how to find and verify candidates,provider_optionsper provider, keys,limits,on_rate_limitand lanes. Read at steps 1 to 8.references/integrations/openrouter-model-selection.md: the aggregator page: catalogue and upstream fields, routing, reasoning on OpenRouter. Read at step 6.references/engine/check-providers.md,references/engine/check-shapes.md: the two probes and what they do not prove. Read at step 10.references/reference/agents.md,references/reference/project.md: every key of an agent and ofaqven.yaml. Read before editing either file.references/reference/provider-catalog.md: built-in providers, their key variables and extras. Read at step 4.references/concepts/what-happens-when-a-model-is-called.md: outcomes and their policies, truncation on reasoning models. Read at step 5.references/mcp-cli/runs.md:run_startwithdataset_item_idandagent_overrides, the node id it takes and its errors. Read at step 9.