ai-integration-verification
Use to verify AI-backed product behavior, provider routing, fallback, tool calls, streaming, structured output, quotas, or model changes; require live contract evidence when possible plus deterministi
SKILL.md.agents/skills/ai-integration-verification/SKILL.md
GitHubAI Integration Verification
Read docs/AI_INTEGRATION_TESTING.md for the full matrix.
- Define the product contract independently of provider wording: required modality, schema, tool behavior, latency/error UX, and safety/privacy constraints.
- Test router logic deterministically with fake provider adapters for 401/403/404/429/500/503/504/timeout/stream interruption.
- Verify retry budget, circuit breaker recovery, same-project quota grouping, and fallback order. Record this deterministic suite as
ai-failure-matrixevidence. - Verify a fallback cannot silently drop image/audio/video/tool/schema requirements.
- Make tool/side-effect execution idempotent; retrying a model turn must not duplicate writes/charges/messages.
- Test secret redaction and confirm keys are absent from client bundles/logs.
- When authorized credentials are available, run a minimal live contract call through the same application integration path and record it as
ai-contractevidence. - If live access is unavailable, leave
ai-contractblocked/external-pending; mocks cannot prove current upstream compatibility. - Do not load-test free upstream APIs to discover limits; use provider documentation plus local simulators.