adaptive-fastopd
Adaptive FastOPD methodology for efficient on-policy distillation with progress-aware rollout horizon expansion. Reduces training time by 49.1-71.2% while maintaining or improving performance by dynam
SKILL.mdcollection/skills/other/adaptive-fastopd/SKILL.md
GitHubAdaptive FastOPD
Adaptive FastOPD is a progress-aware strategy for efficient on-policy distillation (OPD) that dynamically controls rollout horizon expansion based on learning progress signals.
Core Methodology
The key innovation of Adaptive FastOPD is its dual-condition approach for rollout horizon expansion:
- Learning Plateau Detection: Expansion occurs only when learning near the current boundary region has plateaued
- Horizon Utilization Check: Prevents expansion triggered by a small number of long responses
Progress Signals
The learning plateau is determined from four teacher-student signals measured relative to their values upon entering each horizon:
- Relative agreement signals (not absolute thresholds)
- Stage-specific progress responsiveness
- Prevents premature expansion during active learning phases
Benefits
- 49.1-71.2% reduction in training time compared to standard OPD 15K
- Highest average performance across teacher-student pairs
- Robust across hyperparameter settings
- Adaptive to different models and training stages
Implementation Guidelines
When to Use Adaptive FastOPD
Use Adaptive FastOPD when:
- Performing on-policy distillation with online rollout processes
- Computational cost is dominated by long response generation
- Fixed budget or absolute threshold methods are suboptimal
- Training efficiency needs to be maximized without sacrificing performance
Key Parameters to Configure
- Initial rollout horizon: Starting sequence length for student generation
- Progress signal thresholds: Relative change thresholds for the four teacher-student signals
- Horizon utilization ratio: Minimum fraction of batch using current horizon before considering expansion
- Maximum horizon: Upper bound to prevent unbounded expansion
Integration Steps
- Monitor teacher-student signals: Track agreement metrics relative to horizon entry values
- Implement plateau detection: Compare current signals to entry baseline with configured thresholds
- Check horizon utilization: Ensure sufficient batch coverage before expansion
- Expand horizon incrementally: Increase rollout length when both conditions are met
- Reset signal baselines: Update entry values when horizon changes
Pitfalls and Considerations
Common Issues
- Overly sensitive thresholds: May cause frequent horizon changes; start with conservative values
- Insufficient utilization check: Can lead to expansion driven by outliers
- Ignoring model differences: Different teacher-student pairs may need different configurations
Best Practices
- Start with default parameters: Use paper-recommended settings as baseline
- Monitor training curves: Verify that expansion correlates with actual learning plateaus
- A/B test configurations: Compare against fixed-horizon baselines
- Log expansion events: Track when and why horizons change for debugging
Activation Keywords
- adaptive-fastopd
- progress-aware rollout
- efficient on-policy distillation
- dynamic horizon expansion
- teacher-student agreement signals
References
- Original paper: arXiv:2607.29494 "Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation"
- Related work: On-policy distillation (OPD), rollout optimization, teacher-student learning