ffcv
Use FFCV to convert datasets into .beton files, build fast PyTorch loader pipelines, handle image and custom fields/transforms, and tune cache, traversal, and throughput behavior.
SKILL.mdskills/repositories/repo-skills/ffcv/SKILL.md
GitHubFFCV operating router
Use this skill when a task names FFCV, .beton, DatasetWriter, Loader,
OrderOption, FFCV fields/decoders/transforms, or an FFCV cache/throughput
problem. FFCV is a compiled, PyTorch-oriented data storage and loading system;
it replaces the data input path, not the model or optimizer.
Choose a route
- Convert or inspect a dataset — read
dataset-writing. It covers indexable
datasets, local WebDataset shards, field schemas, custom fields,
.betonread-back, and writer resource controls. - Build or debug a loader/pipeline — read loader-pipelines. It covers decoder selection, operation order, images, CPU/GPU boundaries, custom operations, subsets, partial batches, and training integration.
- Diagnose speed, memory, or distributed behavior — read performance-tuning. It covers OS versus process cache, sequential/random/quasi-random traversal, concurrency, compilation, measurement, and distributed constraints.
For a task spanning routes, write the dataset first, validate it structurally,
then construct the loader, then tune one resource variable at a time. Do not
start with benchmark numbers or a full training run when a tiny .beton fixture
can expose the same correctness issue.
Minimal operating contract
- Install FFCV with a compatible PyTorch, NumPy/Numba, OpenCV, and TurboJPEG
runtime. The package includes a native extension; verify both
import ffcvandimport ffcv._libffcv. Read installation before diagnosing compiled-install failures. - A writer mapping's insertion order is the sample tuple order. Declare short, unique ASCII field names and adapt mapping-shaped samples to an explicit tuple before multiprocessing.
- A loader pipeline starts with the field decoder. Keep native NumPy/JIT
transforms before
ToTensor; putToDevicebefore GPU work andToTorchImagewhere the model needs BCHW layout. - Make shape, dtype, device, field names, order, seed, cache mode, batch size,
and
drop_lastexplicit. Validate a complete batch and a one-row tail when partial batches are allowed. - Treat CUDA as additive runtime coverage unless the downstream task truly requires it. A CPU import or smoke test does not prove GPU transfer, normalization, or multi-GPU correctness.
Use api-reference for verified signatures and
object relationships, and troubleshooting for
cross-cutting install, schema, pipeline, cache, and device failures. The small
helper scripts/ffcv_smoke.py creates a temporary
integer .beton file and checks a sequential loader; pass --cuda only on a
reserved compatible GPU.
Scope boundary
This graph is self-contained operating knowledge for package users. It does not require the original checkout, source examples, downloaded datasets, benchmark- scale matrices, ImageNet/CIFAR training, release automation, Docker files, or binary documentation assets. Those are evidence behind the distilled guidance, not runtime dependencies.