cv-gradient-accumulation
Accumulates gradients over multiple mini-batches before stepping the optimizer, simulating larger effective batch sizes.
SKILL.mdskills/cv/gradient-accumulation/SKILL.md
GitHubGradient Accumulation
Overview
When GPU memory limits batch size, accumulate gradients over N mini-batches before calling optimizer.step(). The effective batch size becomes mini_batch × N. This is critical for large image models (ResNet200d, EfficientNet-B7) where batch size 4-8 per GPU would produce noisy gradients.
Quick Start
import torch
from torch.cuda.amp import autocast, GradScaler
accum_steps = 4 # effective batch = mini_batch * 4
scaler = GradScaler()
for step, (images, labels) in enumerate(train_loader):
images, labels = images.cuda(), labels.cuda()
with autocast():
logits = model(images)
loss = criterion(logits, labels) / accum_steps # scale loss
scaler.scale(loss).backward()
if (step + 1) % accum_steps == 0:
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad()
Workflow
- Choose accumulation steps N (effective_batch = mini_batch × N)
- Divide loss by N before backward (keeps gradient scale consistent)
- Call backward every step (gradients accumulate in
.grad) - Call optimizer.step + zero_grad every N steps
- Works seamlessly with mixed precision (GradScaler)
Key Decisions
- Divide loss by N: Essential — without this, accumulated gradients are N× too large
- Learning rate: Keep same as if training with full effective batch size
- BatchNorm: Stats are computed per mini-batch, not effective batch — may differ slightly
- Scheduler step: Step scheduler every N steps or per epoch, not every mini-batch
- Typical N: 2-8; beyond 8, diminishing returns and BN statistics diverge
References
- RANZCR CLiP - Catheter and Line Position Challenge (Kaggle)
- Source: ranzcr-resnext50-32x4d-starter-training