cv-mixed-precision-training
Uses PyTorch AMP autocast and GradScaler for FP16 training, halving memory usage and speeding up training on modern GPUs.
SKILL.mdskills/cv/mixed-precision-training/SKILL.md
GitHubMixed Precision Training
Overview
Run forward pass in FP16 (half precision) while keeping master weights in FP32. This halves GPU memory for activations and enables larger batch sizes or image resolutions. PyTorch's autocast handles dtype selection per operation; GradScaler prevents underflow in FP16 gradients.
Quick Start
import torch
from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
for images, labels in train_loader:
images, labels = images.cuda(), labels.cuda()
optimizer.zero_grad()
with autocast():
logits = model(images)
loss = criterion(logits, labels)
scaler.scale(loss).backward()
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
scaler.step(optimizer)
scaler.update()
Workflow
- Create
GradScaler()before training loop - Wrap forward pass + loss in
autocast()context - Call
scaler.scale(loss).backward()instead ofloss.backward() - Optionally unscale before gradient clipping
scaler.step(optimizer)andscaler.update()replaceoptimizer.step()
Key Decisions
- Memory savings: ~40-50% reduction in activation memory → can double batch size
- Speed: 1.5-2x faster on Volta/Ampere GPUs (V100, A100, RTX 30xx+)
- Accuracy: Virtually identical to FP32 — safe for almost all CV tasks
- Gradient clipping: Must
unscale_beforeclip_grad_norm_to clip in FP32 space - When to skip: Very small models where memory isn't a bottleneck
References
- RANZCR CLiP - Catheter and Line Position Challenge (Kaggle)
- Source: ranzcr-resnext50-32x4d-starter-training