benchmark-ollama-llm
Profile Ollama LLM prefill/decode TPS across short text, long prompts, and vision+text inputs. Handles API routing, metric extraction, and result interpretation.
SKILL.mdbenchmark-ollama-llm/SKILL.md
GitHubBenchmark Ollama LLM Performance
Profile an Ollama model's prefill and decode throughput (TPS), token counts, and latency.
Steps
- Write a Python script using
requeststo hithttp://localhost:11434. - Route API endpoints correctly:
- Text-only prompts →
/api/generate - Vision/multimodal prompts →
/api/chat
- Text-only prompts →
- Extract timing metrics from the JSON response:
prompt_eval_duration(prefill time)eval_duration(decode time)prompt_eval_count(prefill tokens)eval_count(decode tokens)
- Calculate TPS:
count / (duration / 1e9). If duration is0, report0 TPS. - Run benchmarks across categories: short text, long prompt, vision+text.
Interpretation & Pitfalls
- Decode TPS: Stable metric. Expect ~80-120 TPS for 8B models on consumer iGPUs.
- Prefill TPS inflation: Long prompts often show artificially high prefill TPS (e.g., 100k+). Ollama caches prompt embeddings; the API returns near-zero
prompt_eval_durationfor cached prompts, inflating the calculated TPS. Always treat short-text prefill (~1k-3k TPS) as the accurate baseline. - Vision+Text failures: The
/api/chatendpoint for some multimodal models omitsprompt_eval_durationandeval_durationin the JSON response. If timing stats are missing, reportN/Aand note the endpoint limitation. Verify the model actually supports vision (ollama list). - API Key/Billing: Not required for local Ollama. Ensure Ollama is running and serving the target model (
ollama ps).
Example Python Snippet
import requests, time
url = "http://localhost:11434/api/generate" # or /api/chat
payload = {"model": "mymodel", "prompt": "test", "stream": False}
r = requests.post(url, json=payload).json()
pe_d = r.get("prompt_eval_duration", 0)
ev_d = r.get("eval_duration", 0)
pe_c = r.get("prompt_eval_count", 0)
ev_c = r.get("eval_count", 0)
prefill_tps = pe_c / (pe_d / 1e9) if pe_d > 0 else 0
decode_tps = ev_c / (ev_d / 1e9) if ev_d > 0 else 0