adaptive-fastopd

Adaptive FastOPD methodology for efficient on-policy distillation with progress-aware rollout horizon expansion. Reduces training time by 49.1-71.2% while maintaining or improving performance by dynam

SKILL.md

GitHub

Adaptive FastOPD

Adaptive FastOPD is a progress-aware strategy for efficient on-policy distillation (OPD) that dynamically controls rollout horizon expansion based on learning progress signals.

Core Methodology

The key innovation of Adaptive FastOPD is its dual-condition approach for rollout horizon expansion:

  1. Learning Plateau Detection: Expansion occurs only when learning near the current boundary region has plateaued
  2. Horizon Utilization Check: Prevents expansion triggered by a small number of long responses

Progress Signals

The learning plateau is determined from four teacher-student signals measured relative to their values upon entering each horizon:

  • Relative agreement signals (not absolute thresholds)
  • Stage-specific progress responsiveness
  • Prevents premature expansion during active learning phases

Benefits

  • 49.1-71.2% reduction in training time compared to standard OPD 15K
  • Highest average performance across teacher-student pairs
  • Robust across hyperparameter settings
  • Adaptive to different models and training stages

Implementation Guidelines

When to Use Adaptive FastOPD

Use Adaptive FastOPD when:

  • Performing on-policy distillation with online rollout processes
  • Computational cost is dominated by long response generation
  • Fixed budget or absolute threshold methods are suboptimal
  • Training efficiency needs to be maximized without sacrificing performance

Key Parameters to Configure

  1. Initial rollout horizon: Starting sequence length for student generation
  2. Progress signal thresholds: Relative change thresholds for the four teacher-student signals
  3. Horizon utilization ratio: Minimum fraction of batch using current horizon before considering expansion
  4. Maximum horizon: Upper bound to prevent unbounded expansion

Integration Steps

  1. Monitor teacher-student signals: Track agreement metrics relative to horizon entry values
  2. Implement plateau detection: Compare current signals to entry baseline with configured thresholds
  3. Check horizon utilization: Ensure sufficient batch coverage before expansion
  4. Expand horizon incrementally: Increase rollout length when both conditions are met
  5. Reset signal baselines: Update entry values when horizon changes

Pitfalls and Considerations

Common Issues

  • Overly sensitive thresholds: May cause frequent horizon changes; start with conservative values
  • Insufficient utilization check: Can lead to expansion driven by outliers
  • Ignoring model differences: Different teacher-student pairs may need different configurations

Best Practices

  • Start with default parameters: Use paper-recommended settings as baseline
  • Monitor training curves: Verify that expansion correlates with actual learning plateaus
  • A/B test configurations: Compare against fixed-horizon baselines
  • Log expansion events: Track when and why horizons change for debugging

Activation Keywords

  • adaptive-fastopd
  • progress-aware rollout
  • efficient on-policy distillation
  • dynamic horizon expansion
  • teacher-student agreement signals

References

  • Original paper: arXiv:2607.29494 "Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation"
  • Related work: On-policy distillation (OPD), rollout optimization, teacher-student learning