cv-pack-padded-sequence-loss
Uses pack_padded_sequence to exclude padding tokens from cross-entropy loss in variable-length sequence generation.
SKILL.mdskills/cv/pack-padded-sequence-loss/SKILL.md
GitHubPack Padded Sequence Loss
Overview
In image captioning and seq2seq tasks, target sequences have different lengths and are padded. Computing cross-entropy on padding tokens wastes compute and leaks gradient noise. pack_padded_sequence strips out padding positions before loss computation, giving a clean loss signal from real tokens only.
Quick Start
from torch.nn.utils.rnn import pack_padded_sequence
# Sort batch by decreasing sequence length (required by pack_padded_sequence)
caption_lengths, sort_ind = caption_lengths.squeeze(1).sort(dim=0, descending=True)
predictions = predictions[sort_ind]
targets = targets[sort_ind]
decode_lengths = (caption_lengths - 1).tolist() # exclude <sos>
# Pack to strip padding, then compute loss on real tokens only
packed_preds = pack_padded_sequence(predictions, decode_lengths, batch_first=True).data
packed_targets = pack_padded_sequence(targets, decode_lengths, batch_first=True).data
loss = nn.CrossEntropyLoss()(packed_preds, packed_targets)
Workflow
- Sort batch by sequence length (descending)
- Compute
decode_lengths= actual length of each target sequence - Pack both predictions and targets with
pack_padded_sequence - Compute cross-entropy on
.dataattribute (flattened real tokens) - Backpropagate — gradients only flow through non-padding positions
Key Decisions
- Sort requirement:
pack_padded_sequencerequires descending sort; re-sort encoder outputs too - Alternative: Use
ignore_indexin CrossEntropyLoss for padding token ID — simpler but slightly less efficient - Combine with: Teacher forcing during training, greedy/beam search during inference
- Metrics: Compute Levenshtein distance or BLEU on unpacked decoded sequences