p3o_surrogate_loss
Build decomposed P3O surrogate loss terms with clipped replay gradients and behavior-to-target KL regularization.
SKILL.mdbasicagent-skill/skill_library/paperbench/sapg/skill/p3o_policy_on_off/p3o_surrogate_loss/SKILL.md
GitHubP3O Surrogate Loss
Use this skill when a recovery or implementation must preserve the P3O objective from Equation 14. It is not a full RL framework; it computes deterministic scalar components that can be embedded in a learner.
Inputs
- On-policy score-gradient proxies and advantages.
- Off-policy score-gradient proxies, advantages, and importance ratios.
- Target and behavior action distributions for KL estimation.
clip_thresholdandkl_coefficient, usually from the ESS scheduler.
Outputs
on_policy,off_policy,kl_penalty, andobjectivescalar values.- Clipped ratios used for replay data.
Workflow
- Multiply on-policy advantages by score-gradient proxies.
- Clip each replay importance ratio at
cand multiply by replay advantage and score proxy. - Compute
KL(beta || pi_theta)over behavior and target distributions. - Combine terms as
on + off - lambda * KL.
Source Boundary
Use this skill with the paper, module documents, generated artifacts, and ordinary package documentation. Do not read or depend on the original P3O repository.
Validation
Run python scripts/<script>.py --self-test or python -m pytest tests from the skill directory. The bundled tests use only the Python standard library.