nlp-worker-stratified-kfold
Stratifies CV folds by annotator/worker ID to prevent annotator style leakage across train and validation splits in crowd-sourced datasets.
SKILL.mdskills/nlp/worker-stratified-kfold/SKILL.md
GitHubWorker-Stratified KFold
Overview
In crowd-sourced annotation datasets, multiple workers label different subsets of examples. If the same worker's annotations appear in both train and validation, the model can learn annotator-specific biases rather than the task signal. Stratifying folds by worker ID ensures each annotator's labels stay entirely within one fold, giving a more honest estimate of generalization. This is distinct from group-based splitting — here we stratify (balance worker distribution) rather than group (isolate entire groups).
Quick Start
from sklearn.model_selection import StratifiedKFold
# df has columns: 'text', 'label', 'worker' (annotator ID)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
df['fold'] = -1
for fold, (_, val_idx) in enumerate(skf.split(X=df, y=df['worker'])):
df.loc[val_idx, 'fold'] = fold
# Train/val split for a specific fold
train_df = df[df['fold'] != 0]
val_df = df[df['fold'] == 0]
Workflow
- Identify the worker/annotator ID column in the dataset
- Use StratifiedKFold with
y=worker_idto balance workers across folds - Each fold gets a proportional share of each worker's annotations
- Train and evaluate per fold as usual
Key Decisions
- Stratify vs Group: Stratify balances workers across folds; GroupKFold isolates workers entirely — choose based on whether workers overlap with test set
- Many workers: If workers >> folds, stratification is approximate; consider GroupKFold instead
- vs label stratification: Can combine both by creating a composite key
f"{worker}_{label}" - When to use: Any dataset with annotator IDs — toxicity, NER, sentiment with crowd labels