llamacpp
Complete llama.cpp C/C++ API reference covering model loading, inference, text generation, embeddings, chat, tokenization, sampling, batching, KV cache, LoRA adapters, and state management. Triggers o
SKILL.mdplugins/llamacpp/skills/llamacpp
GitHubllama.cpp C API Guide
Comprehensive reference for the llama.cpp C API, documenting all non-deprecated functions and common usage patterns.
Overview
llama.cpp is a C/C++ implementation for LLM inference with minimal dependencies and state-of-the-art performance. This skill provides:
- Complete API Reference: All non-deprecated functions organized by category
- Common Workflows: Working examples for typical use cases
- Best Practices: Patterns for efficient and correct API usage
Quick Start
See references/workflows.md for complete working examples. Basic workflow:
llama_backend_init()- Initialize backendllama_model_load_from_file()- Load modelllama_init_from_model()- Create contextllama_tokenize()- Convert text to tokensllama_decode()- Process tokensllama_sampler_sample()- Sample next token- Cleanup in reverse order
When to Use This Skill
Use this skill when:
- API Lookup: You need to find a specific function (e.g., "How do I load a model?", "What function creates a context?")
- Code Generation: You're writing C code that uses llama.cpp
- Workflow Guidance: You need to understand the steps for a task (e.g., text generation, embeddings, chat)
- Advanced Features: You're working with batches, sequences, LoRA adapters, state management, or custom sampling
- Migration: You're updating code from deprecated functions to current API
Core Concepts
Key Objects
llama_model: Loaded model weights and architecturellama_context: Inference state (KV cache, compute buffers)llama_batch: Input tokens and positions for processingllama_batch_ext: Opaque extended batch builder, processed byllama_process()(b11284+)llama_sampler: Token sampling configurationllama_vocab: Vocabulary and tokenizerllama_memory_t: KV cache memory handle
Typical Flow
- Initialize:
llama_backend_init() - Load Model:
llama_model_load_from_file() - Create Context:
llama_init_from_model() - Tokenize:
llama_tokenize() - Process:
llama_encode()orllama_decode() - Sample:
llama_sampler_sample() - Generate: Repeat steps 5-6
- Cleanup: Free in reverse order
API Reference
For detailed API documentation, the complete API is split across 6 files for efficient targeted loading. Start with references/api-core.md which links to all other sections.
API Files:
- api-core.md (373 lines) - Initialization, parameters, model loading, quantization structs
- api-model-info.md (241 lines) - Model properties, architecture detection, metadata enums
- api-context.md (423 lines) - Context, memory (KV cache), state management
- api-inference.md (507 lines) - Batch operations (incl. extended batch API), inference, tokenization, chat
- api-sampling.md (524 lines) - All 20+ sampling strategies (incl. adaptive-p) + backend sampling API
- api-advanced.md (426 lines) - LoRA adapters, performance, training, constants, C++ RAII wrappers
Total: 219 active functions (b11526) across 6 organized files
Quick Function Lookup
Most common: llama_backend_init(), llama_model_load_from_file(), llama_init_from_model(), llama_tokenize(), llama_decode(), llama_sampler_sample(), llama_vocab_is_eog(), llama_memory_clear()
Extended batch (b11284+): llama_batch_ext_init(), llama_batch_ext_add_token(), llama_batch_ext_set_pos(), llama_batch_ext_set_output_logits(), llama_process()
See references/api-core.md for the full API index linking to all function signatures.
Common Workflows
See references/workflows.md for 16 complete working examples: basic text generation, chat, embeddings, batch processing, multi-sequence, LoRA, state save/load, custom sampling (XTC/DRY), encoder-decoder models, model detection, memory management patterns, streaming, and the extended batch API (llama_process).
Best Practices
See references/workflows.md for detailed best practices. Key points:
- Always use default parameter functions (
llama_model_default_params(), etc.) - Check return values for errors
- Free resources in reverse order of creation
- Handle dynamic buffer sizes for tokenization
- Query actual context size after creation (
llama_n_ctx()) - Check for end-of-generation with
llama_vocab_is_eog()
Common Patterns
End-of-generation check (llama_vocab_is_eog()), logits retrieval (llama_get_logits_ith()), batch creation (llama_batch_get_one()), tokenization buffer handling. See references/workflows.md for complete code examples.
Troubleshooting
Common Issues
Model loading fails:
- Verify file path and GGUF format validity
- Check available RAM/VRAM for model size
- Reduce
n_gpu_layersif GPU memory insufficient
Tokenization returns negative value:
- Buffer too small; reallocate with
-nsize and retry - See tokenization pattern in Common Patterns
Decode/encode returns non-zero:
- Verify batch initialization (
llama_batch_get_one()orllama_batch_init()) - Check context capacity (
llama_n_ctx()) - Ensure positions within context window
Silent failures / no output:
- Check if
llama_vocab_is_eog()immediately returns true - Verify sampler initialization
- Enable logging:
llama_log_set()
Performance issues:
- Increase
n_threadsfor CPU - Set
n_gpu_layersfor GPU offloading - Use larger
n_batchfor prompts - See Performance & Utilities
Sliding Window Attention (SWA) issues:
- If using Mistral-style models with SWA, set
ctx_params.swa_full = trueto access beyond attention window - Check:
llama_model_n_swa(model)to detect SWA size and configuration needs - Symptoms: Token positions beyond window size causing decode errors
Per-sequence state errors:
- Ensure sequence ID matches when loading:
llama_state_seq_load_file(ctx, "file", dest_seq_id, ...) - Verify token buffer is large enough for loaded tokens
- Check sequence wasn't cleared or removed before loading state
Model type detection:
- Use
llama_model_has_encoder()before assuming decoder-only architecture - For recurrent models (Mamba/RWKV), KV cache behavior differs from standard transformers
- Encoder-decoder models require
llama_encode()thenllama_decode()workflow
For advanced issues: https://github.com/ggerganov/llama.cpp/discussions
Resources
- API Reference (6 files, 2,494 lines total) - Complete API reference split by category for targeted loading:
- api-core.md - Initialization, parameters, model loading, quantization structs
- api-model-info.md - Model properties, architecture detection, metadata enums
- api-context.md - Context, memory, state management
- api-inference.md - Batch (incl.
llama_batch_ext), inference, tokenization, chat - api-sampling.md - All 20+ sampling strategies (incl. adaptive-p) + backend sampling API
- api-advanced.md - LoRA, performance, training, constants, C++ RAII (
llama-cpp.h)
- references/workflows.md (1,706 lines) - 16 complete working examples: basic workflows (text generation, chat, embeddings, batching, sequences), intermediate (LoRA, state, sampling, encoder-decoder, memory), advanced features (XTC/DRY, per-sequence state, model detection), production applications (interactive chat, streaming), and the extended batch API.
What's New in b11526
b11526 (242 commits since b11284) — no functions added, removed, or re-signatured. Changes are data/struct-only:
BREAKING (data, not code):
LLAMA_SESSION_VERSION10 → 11 andLLAMA_STATE_SEQ_VERSION3 → 4 (KV state now saves exact rotation metadata). Session/sequence-state files from older builds are rejected on load — regenerate them. See api-context.md.
New:
llama_context_params.moe_cache_size(size_t, [EXPERIMENTAL]) — device cache in bytes for MoE experts kept in host memory, split across devices like the layers;0= disabled.LLAMA_VOCAB_TYPE_PLAMO3 = 8(PLaMo-3 tokenizer).
Previously (b11284, 164 commits since b11120) — additive, non-breaking: 14 new functions, nothing removed, no signature changes.
- New extended batch API — opaque
struct llama_batch_ext+llama_process(ctx, type, batch)withenum llama_process_type(LLAMA_PROCESS_TYPE_ENCODE/DECODE); return codes matchllama_decode(). Builder:llama_batch_ext_init(ctx)/_free/_clear,_add(batch, seq_id)/_add_token/_add_embd(return batch index, or-1full /-2invalid token /-3invalid seq id),_add_seq,_set_embd_token,_set_embd_state(stub),_set_output_embd/_set_output_logits(equivalent for now),_set_pos(caller must set positions; M-RoPE embedding entries take multiple positions). Newstruct llama_embd { data, n_rows, n_embd }. batch_ext get_logits/get_embeddings are TODO — read outputs withllama_get_logits_ith(ctx, idx).llama_decode()now convertsllama_batchto this internally. See api-inference.md and workflow 16. - C++:
llama_batch_ext_ptr(+llama_batch_ext_deleter) inllama-cpp.h. - New:
llama_get_causal_attn(ctx)— getter paired withllama_set_causal_attn(). - Heads-up:
llama_numa_init()marked "TODO: deprecate and make part ofllama_backend_init()" (not deprecated yet).
Previously (b11120, 252 commits since b10868) — small, non-breaking C API changes:
- New:
llama_adapter_lora_init_from_file_ptr(model, FILE *)— load a LoRA adapter from an openFILE*at its current position (mirrorsllama_model_load_from_file_ptr). See api-advanced.md. - GGUF-in-a-larger-file:
llama_model_load_from_file_ptr()reads from the current file position; the GGUF data section is now aligned relative to the GGUF start. For mmap, its absolute offset must be 32-byte aligned. llama_sampler_chain_n()now returnsint32_t(wasint) — ABI-identical on all mainstream platforms.- New
LLAMA_VOCAB_TYPE_TEST = 7(dummy tokenizer for tests).
Previously (b10868, 203 commits since b10665) — no functions added or removed; the only public C API change is a rename:
BREAKING (rename, not behavior):
enum llama_tensor_read_lazy→enum llama_lazy_mode; its constantsLLAMA_TENSOR_READ_LAZY_OFF/AUTO/ON→LLAMA_LAZY_MODE_OFF/AUTO/ON; andllama_model_params.tensor_read_lazy→llama_model_params.lazy_mode. Behavior is unchanged — update call sites setting the old field/enum names. See api-core.md.
Previously (b10665, 249 commits since b10416) — no functions added, removed, or re-signatured. The public C API changed only in struct fields, one new enum, and the state-file versions:
BREAKING (data, not code):
LLAMA_SESSION_VERSION9 → 10 andLLAMA_STATE_SEQ_VERSION2 → 3 (recurrent-state rollback inggml_ssm_scanchanged the serialized layout). Session/sequence-state files from older builds are rejected —llama_state_load_file()/llama_state_seq_load_file()fail on them. Regenerate cached sessions; check the return value and fall back to re-ingesting the prompt.
New lazy tensor reading (renamed in b10868, see above):
- Added
enum llama_tensor_read_lazy(LLAMA_TENSOR_READ_LAZY_OFF/AUTO/ON) +llama_model_params.tensor_read_lazy. Faults in rows of arch-marked tensors on demand instead of reading them whole at load — cuts resident memory and load latency for models with very large sparsely-used tensors.AUTOapplies only to marked tensors > 4 GiB; bothAUTOandONrequire an mmap load mode. See api-core.md.
New quantizer memory cap:
llama_model_quantize_params.max_buf_size(size_t) — max bytes of tensor rows held in memory at once,0= default (8 GiB). Lower it to quantize very large models on memory-constrained machines.
Previously (b10416, 158 commits since b10258) — multi-output backend sampling, versioning, and a DRY sampler signature change:
BREAKING:
llama_sampler_init_dry()no longer takes theint32_t n_ctx_trainparameter (previously the 2nd argument, right aftervocab). Update all call sites — see api-sampling.md.
New multi-output backend sampling [EXPERIMENTAL]:
llama_context_params.n_outputs_max_per_seq(uint32_t) — max sampled outputs per sequence in a ubatch (0 =n_outputs_max).llama_sampler_i.backend_init()gained auint32_t n_outputs_max_per_seqparameter; newbackend_reset()/copy_state()callbacks for authors of custom backend samplers.llama_sampler_copy()— copy mutable sampler state between two same-type/config samplers without losing the destination's bound compute graph.llama_get_sampled_token_ith(): with multiple outputs, sampler state now advances on token acceptance, not on read; accept a contiguous prefix in output order (no gaps).
New load-mode default:
LLAMA_LOAD_MODE_AUTO(-1) is the new default forllama_model_params.load_mode(wasLLAMA_LOAD_MODE_MMAP). Auto-detects based on device capabilities — e.g. avoids mmap on iGPUs.
New:
llama_version()— returns the llama.cpp library version string (from CMake's new semantic-versioning setup).
Doc correction (no code change): llama_sampler_init_penalties()'s penalty_last_n doc previously said "-1 = context size"; that was never accurate — negative values are clamped to 0 (disabled). History-based samplers (DRY, penalties) no longer resolve a "full-context window" from training context size, since backend sampling constructs samplers before a llama_context (and its resolved context length) exists.
Not in llama.h/llama-cpp.h, mentioned for awareness: mtmd gained Qwen3-TTS support — a breaking change to the llama-tts CLI binary (outside this skill's C API scope).
Previously (b10258, 183 commits since b10075):
BREAKING:
llama_sampler_init_penalties()gained a new required first parametern_vocab(source it viallama_vocab_n_tokens(vocab)). Update all call sites.
New load-mode API (replaces three booleans):
enum llama_load_mode(LLAMA_LOAD_MODE_NONE/MMAP/MLOCK/MMAP_MLOCK/DIRECT_IO) +llama_load_mode_name()/llama_load_mode_from_str().llama_model_params.load_modereplaces the removeduse_mmap,use_direct_io, anduse_mlockboolean fields.llama_model_params.load_mtp(bool) — whether to load MTP layers.
New vocab function:
llama_vocab_get_suppress_tokens()— model-specific suppress tokens (gguf keytokenizer.ggml.suppress_tokens).
Previously (b10075, ~205 commits since b9870): touched the public C API in exactly one place — new
LLAMA_FTYPE_MOSTLY_Q2_0 = 41 quantization type (CPU backend), PR #24448. Everything else in that range (new
Hy3/hy_v3 model + MTP speculative decoding, server reasoning_budget_tokens, mtmd NUL-truncation fix,
deepseek-ocr v1 multi-tile, /responses streaming timings, etc.) was internal/server-side and did not change
llama.h.
Recent (added in b9859→b9870, PR #25134):
llama_model_ftype()— returns the model's file type as anenum llama_ftype(e.g.LLAMA_FTYPE_MOSTLY_Q8_0).llama_ftype_name()— converts anenum llama_ftypeto a human-readable string (e.g."Q8_0","Q4_K - Medium"). Pair the two to display a loaded model's quantization.
Recent (added in b9840, from b9704):
llama_model_n_layer_nextn()— returns the number of NextN (Multi-Token Prediction / MTP) layers in the model. These speculative next-token prediction layers power MTP-capable architectures such as DeepSeek V3/V4, GLM, Qwen3.5-MoE, and Step3.5. Returns 0 for non-MTP models. The total layer count equalsllama_model_n_layer()(effective layers) + this value.
Recent (added in b9704) [EXPERIMENTAL]:
llama_context_params.n_outputs_max(uint32_t) — max outputs in a ubatch (0 =n_batch). Cap it to reserve less output VRAM when you read only a few logits/embeddings per batch (e.g. one output per sequence during generation).llama_context_params.ctx_other(struct llama_context *) — a source/target/parent context for sharing inference results orllama_memory(KV cache) between two contexts; used by MTP setups such as Gemma4 MTP.llama_set_warmup()deprecated — perform warmup runs manually instead. It changed graph topology with MoE models (causing extra reallocations) and will be removed in a future release.
Still recent (added in b9246) [EXPERIMENTAL]:
- Multi-Token Prediction:
enum llama_context_type(LLAMA_CONTEXT_TYPE_DEFAULT/LLAMA_CONTEXT_TYPE_MTP) +llama_context_params.ctx_type - Recurrent-state rollback (Mamba/RWKV):
llama_context_params.n_rs_seq+llama_n_rs_seq(ctx) - Sequence-state flags:
LLAMA_STATE_SEQ_FLAGS_NONE(0),LLAMA_STATE_SEQ_FLAGS_ON_DEVICE(2)
Stable Since b8809:
- Model loading:
llama_model_load_from_file_ptr(),llama_model_init_from_user() - Quantization types: MXFP4_MOE (38), NVFP4 (39), Q1_0 (40)
- Split mode:
LLAMA_SPLIT_MODE_TENSOR(3) for backend-agnostic tensor parallelism - Backend sampling API (EXPERIMENTAL): GPU-accelerated sampling via context params
- Adaptive-P sampler:
llama_sampler_init_adaptive_p()
Key Differences from Deprecated API
If you're updating old code:
- Use
llama_model_load_from_file()instead ofllama_load_model_from_file() - Use
llama_model_free()instead ofllama_free_model() - Use
llama_init_from_model()instead ofllama_new_context_with_model() - Use
llama_vocab_*()functions instead ofllama_token_*() - Use
llama_state_*()functions instead of deprecated state functions - Use
llama_set_adapters_lora()instead of the removedllama_set_adapter_lora()for LoRA adapters - Use
llama_vocab_bos()instead ofllama_vocab_cls()(CLS is equivalent to BOS) - Use
llama_sampler_init_grammar_lazy_patterns()instead ofllama_sampler_init_grammar_lazy() - Perform warmup runs manually instead of calling deprecated
llama_set_warmup()(deprecated in b9704)
See the API reference for complete mappings.