drpo-llm-rl

Divergence Regularized Policy Optimization (DRPO) — smooth advantage-weighted quadratic regularizer replacing hard trust-region masks in LLM reinforcement learning. Use when optimizing LLM post-traini

SKILL.md

GitHub

DRPO for LLM RL

Description

Divergence Regularized Policy Optimization (DRPO) methodology for stable and efficient reinforcement learning fine-tuning of large language models. Replaces the hard trust-region mask of DPPO with a smooth advantage-weighted quadratic regularizer on policy shift, preserving trust-region geometry while inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary.

Activation Keywords

  • DRPO
  • divergence regularized policy optimization
  • trust region LLM RL
  • smooth policy regularization
  • GRPO improvement
  • DPPO alternative
  • 散度正则化策略优化
  • 大模型强化学习优化

Core Concepts

Problem

  • LLM RL is often off-policy due to training-inference mismatch and policy staleness
  • PPO/GRPO use ratio-clipping which is a poor proxy for distributional shift in long-tailed vocabularies
  • DPPO uses divergence-based mask but has a hard boundary: once a token crosses the trust-region boundary, its gradient is discarded rather than corrected

DRPO Solution

  1. Smooth Regularizer: Replaces hard mask with smooth advantage-weighted quadratic regularizer on policy shift
  2. Trust-Region Geometry: Preserves the same trust-region geometry as DPPO
  3. Continuous Gradient Weights: Bounded, continuous weights that attenuate (not discard) diverging updates
  4. Corrective Signals: Provides correction beyond the boundary, not just zeroing out

Mathematical Framework

  • Policy shift regularization: Quadratic penalty weighted by advantage estimates
  • Trust-region: Defined by sampled token's absolute probability shift (like DPPO)
  • Gradient weighting: Smooth function of policy divergence → gradient is scaled, not clipped to zero

Tools Used

  • terminal: Run training scripts, monitor RL convergence
  • file: Write training configurations, analyze loss curves
  • search: Research LLM RL best practices, compare with PPO/GRPO/DPPO

Usage Patterns

Pattern 1: Implementing DRPO in LLM RL Pipeline

When training an LLM with RL fine-tuning (PPO, GRPO variants):

  1. Identify the trust-region mechanism in current policy optimization
  2. Replace ratio-clipping (PPO/GRPO) or hard-mask (DPPO) with DRPO's smooth quadratic regularizer
  3. Compute advantage-weighted policy shift for each token
  4. Apply continuous gradient weighting based on divergence magnitude
  5. Monitor stability metrics: gradient norms, policy divergence, reward curve

Pattern 2: Diagnosing LLM RL Instability

When LLM RL training exhibits:

  • Reward collapse or oscillation
  • Policy degeneration (repetitive outputs)
  • Gradient explosions in long-tailed vocabulary regions Apply DRPO analysis:
  1. Check if ratio-clipping is causing abrupt gradient discontinuities
  2. Measure actual distributional shift vs. importance ratio proxy
  3. Consider DRPO's smooth regularizer as stabilization mechanism

Pattern 3: Comparing Trust-Region Methods

When evaluating RL fine-tuning approaches:

  1. PPO/GRPO: Ratio-clipping, simple but poor proxy for distributional shift
  2. DPPO: Divergence-based mask, better trust-region but hard boundary (gradient discard)
  3. DRPO: Smooth advantage-weighted regularizer, preserves trust-region with continuous corrections

Instructions for Agents

Step 1: Assess Current RL Setup

  • Identify the policy optimization method (PPO, GRPO, DPPO, etc.)
  • Check for trust-region violations during training
  • Measure importance ratio quality vs. actual distributional shift

Step 2: Design DRPO Regularizer

  • Define the quadratic regularization term on policy shift
  • Weight by advantage estimates (higher advantage → more tolerance for shift)
  • Set regularization temperature/hyperparameters

Step 3: Implementation

  • Replace hard mask with smooth regularizer in the loss function
  • Ensure gradient flow is continuous across trust-region boundary
  • Validate that trust-region geometry is preserved

Step 4: Validation

  • Compare reward curves: DRPO should show smoother convergence
  • Check gradient norms: should be bounded, not spiking
  • Verify out-of-boundary behavior: corrections, not zeros

Error Handling

Training Instability Persists

  1. Check regularization temperature — may need adjustment
  2. Verify advantage estimation quality
  3. Compare with DPPO baseline to isolate the smooth vs. hard boundary effect

Gradient Vanishing

  1. Ensure advantage weighting doesn't over-attenuate
  2. Check that the quadratic regularizer preserves sufficient gradient signal
  3. Consider mixing with a small ratio-clipping term as safety net

References

  • arXiv: 2606.09821 — "Rethinking the Divergence Regularization in LLM RL"
  • Related: DPPO (Divergence-based PPO), PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization)

Examples

Example 1: Replacing DPPO Hard Mask with DRPO

# Before (DPPO hard mask):
if divergence > threshold:
    gradient = 0  # Discard

# After (DRPO smooth regularizer):
weight = smooth_function(divergence, advantage)
gradient *= weight  # Attenuate, never zero

Notes

  • DRPO was shown to improve stability and efficiency across model scales, architectures, and precision settings
  • The key insight: hard masks lose information by discarding gradients beyond boundary; smooth regularizers attenuate while preserving signal
  • Particularly beneficial for long-tailed vocabulary distributions where importance ratios are unreliable