grpo

Set up and run reinforcement learning with verifiable rewards (RLVR/GRPO) for math, code, or custom environments using the Tinker API. Use when the user wants to do RL training, GRPO, reward-based opt

SKILL.md

GitHub

Group Relative Policy Optimization (GRPO / RL)

Help the user set up and run RL training with verifiable rewards using the Tinker API.

Step 1: Understand the request

Ask the user (if not already specified):

  • Model: Which model to train (e.g., meta-llama/Llama-3.1-8B-Instruct, Qwen/Qwen3-8B)
  • Environment/Task: What type of reward signal — math (GSM8K, DeepMath, arithmetic), code (DeepCoder), instruction following (IFBench), or custom
  • Reward type: Verifiable (programmatic correctness) or learned (preference model)

Step 2: Reference existing recipes

Read these files for patterns:

  • tinker_cookbook/recipes/rl_basic.py — Minimal RL example (GSM8K)
  • tinker_cookbook/recipes/math_rl/train.py — Full math RL with multiple environments and loss functions
  • tinker_cookbook/recipes/code_rl/train.py — Code generation RL with sandbox execution
  • tinker_cookbook/recipes/rubric/train.py — Rubric-graded RL with LLM scoring
  • tinker_cookbook/rl/train.py — Core RL training loop
  • tinker_cookbook/rl/types.py — Env, EnvGroupBuilder, RLDatasetBuilder
  • docs/rl/rl-basic.mdx — Getting started
  • docs/rl/rl-envs.mdx — Custom environments
  • docs/rl/rl-hyperparams.mdx — Hyperparameter guidance

Step 3: Configure the training run

Environment Setup

RL requires an environment that produces rewards. Key patterns:

Built-in environments:

  • Gsm8kDatasetBuilder — Grade-school math (from recipes/math_rl/math_env.py)
  • ArithmeticDatasetBuilder — Simple arithmetic
  • DeepMathDatasetBuilder, PolarisDatasetBuilder — Advanced math
  • DeepCoderDatasetBuilder — Code generation with sandbox
  • RubricDatasetBuilder — Rubric-graded tasks

Custom environments: Implement the Env protocol from tinker_cookbook/rl/types.py. Key points:

  • Env objects are single-use (no reset method)
  • Create new envs via EnvGroupBuilder each batch
  • Each env returns a float reward

Key Hyperparameters

  • group_size: Number of rollouts per prompt (typically 4-16). Advantages are centered within each group.
  • groups_per_batch (or batch_size): Number of problems per batch
  • max_tokens: Maximum generation length
  • learning_rate: Typically 1e-5 to 4e-5 for RL
  • kl_penalty_coef: KL penalty against reference model (0.0 = no penalty)
  • temperature: Sampling temperature (default 1.0)
  • rollout_error_tolerance: Tolerance for rollout errors (False = crash on any error, True = retry failed trajectories with default budget, RetryOnFailure(max_retries=5) = custom retry budget). Error counts logged as rollout_errors/* metrics.

Loss Functions

  • importance_sampling — Default, on-policy
  • ppo — Proximal Policy Optimization (clipped)
  • cispo — Conservative Importance Sampling PPO
  • dro — Distributionally Robust Optimization
  • Configure via loss_fn and loss_fn_config parameters

Async Training (Off-Policy)

For overlapping sampling and training:

async_config=AsyncConfig(
    max_steps_off_policy=cli_config.max_steps_off_policy,
    groups_per_batch=cli_config.groups_per_batch,
)

Step 4: Write the training script

Follow the pattern from rl_basic.py / math_rl/train.py:

import asyncio
import chz
from tinker_cookbook import cli_utils, model_info
from tinker_cookbook.rl import train

def build_config_blueprint() -> chz.Blueprint[train.Config]:
    model_name = "meta-llama/Llama-3.1-8B-Instruct"
    renderer_name = model_info.get_recommended_renderer_name(model_name)

    # Configure your dataset builder with environment
    builder = ...  # e.g., Gsm8kDatasetBuilder(...)

    return chz.Blueprint(train.Config).apply({
        "model_name": model_name,
        "renderer_name": renderer_name,
        "log_path": "/tmp/tinker-examples/my_rl_run",
        "dataset_builder": builder,
        "learning_rate": 4e-5,
        "max_tokens": 256,
        "eval_every": 20,
    })

def main(config: train.Config):
    cli_utils.check_log_dir(config.log_path, behavior_if_exists="ask")
    asyncio.run(train.main(config))

For the full CLI pattern with @chz.chz config class, see recipes/math_rl/train.py.

Step 5: Run

python -m tinker_cookbook.recipes.<recipe_name>

Override: python -m tinker_cookbook.recipes.<recipe_name> env=gsm8k group_size=16 learning_rate=4e-5

Step 6: Add tests

If you created a new recipe, add a smoke test so CI catches regressions:

# tests/recipes/test_recipe_<name>.py
import pytest
from tests.helpers import run_recipe

@pytest.mark.integration
def test_<recipe_name>():
    run_recipe(
        "tinker_cookbook.recipes.<recipe_name>.train",
        ["behavior_if_log_dir_exists=delete", "groups_per_batch=4", "group_size=2"],
    )

run_recipe() automatically passes max_steps=2 so the recipe runs 2 training steps and exits. For environment logic (reward grading, env setup), add unit tests as *_test.py next to the source:

  • Example: tinker_cookbook/recipes/math_rl/math_env_test.py

Step 7: Export weights (optional)

After training, export weights using the tinker_cookbook.weights API:

from tinker_cookbook import weights

adapter_dir = weights.download(tinker_path="tinker://run-id/sampler_weights/final", output_dir="./adapter")
weights.build_hf_model(base_model="meta-llama/Llama-3.1-8B-Instruct", adapter_path=adapter_dir, output_path="./model")
weights.publish_to_hf_hub(model_path="./model", repo_id="user/my-finetuned-model")

Common pitfalls

  • Env objects are single-use — always create fresh envs via builder
  • Advantages are centered within each group — group_size matters for variance reduction
  • max_tokens too small truncates reasoning; too large wastes compute
  • Start with small groups_per_batch for debugging, scale up for real runs
  • Use num_substeps > 1 for very large batches to split optimizer steps