aiwf-inference-serving
AIWF inference serving skill for local model runtimes. Use for vLLM, llama.cpp server, Ollama, OpenAI-compatible APIs, model endpoint health, auth and network binding, telemetry, queueing, reload and
SKILL.mdskills/aiwf-inference-serving/SKILL.md
GitHubAIWF inference serving
Use this skill when a model needs to be served, kept online, tested through an API, or diagnosed as a local runtime service.
Safety gate
Do not expose a service to the public network, open firewall rules, publish credentials, or start long-running servers unless Shawn explicitly asks. Localhost checks and config review are allowed.
Workflow
- Read project guidance, service scripts, launchers, Docker or WSL notes, and current port conventions.
- Identify the serving runtime:
- vLLM
- llama.cpp server
- Ollama
- Transformers or Diffusers API worker
- FastAPI, Gradio, or project-local OpenAI-compatible wrapper
- Read
references/serving-checklist.mdbefore changing launch commands or endpoint code. - Create a serving plan for non-trivial work:
python <this-skill>\scripts\create_serving_plan.py --runtime vllm --model <model-path> --port 8000 --out <plan.json>
- Verify model loading constraints with
aiwf-model-loaderwhen the server load path depends on dtype, quantization, GGUF metadata, LoRA state, tokenizer, projector, or context length. - Verify service contract:
- bind address and port
- auth or local-only boundary
- health endpoint
- OpenAI-compatible paths when advertised
- model list endpoint
- generation endpoint
- streaming support
- error shape
- Verify runtime operations:
- startup and shutdown
- reload and unload
- queueing and concurrency
- cancellation where supported
- telemetry freshness
- log path and crash visibility
- Verify cheaply first with health probes,
/v1/models, tiny prompt calls, and log inspection. Run load or latency loops only when Shawn asks.
Output
Report:
- runtime and model
- exact launch command or config
- endpoint URLs checked
- local-only or auth boundary
- telemetry and queueing status
- smoke command and result
- remaining stability risks
Use aiwf-ai-evals for benchmark or before/after promotion gates after the service is reachable.