ml-agents
Unity ML-Agents reinforcement learning — Agent lifecycle, observation/action contracts, reward shaping, self-play and ELO, trainer YAML, headless sweeps, and the version handshake. Load when writing A
SKILL.md.claude/skills/ml-agents/SKILL.md
GitHubML-Agents — Reinforcement Learning in Unity
com.unity.ml-agents (C#) trains policies against a Python trainer (mlagents) over
a gRPC handshake. The Unity side collects observations and applies actions; the
Python side owns the optimizer, the checkpoints and the .onnx export.
The version handshake is the first thing to check
The two halves negotiate a communication API version. If those differ the handshake is refused and the trainer exits — often with a message that reads like a port problem.
| Half | Where | This project |
|---|---|---|
| C# | Packages/manifest.json → com.unity.ml-agents |
4.1.0, comms API 1.5.0 |
| Python | .venv → pip show mlagents |
1.1.0, comms API 1.5.0 |
Package version equality is not the requirement — comms API equality is. Upgrading one side alone breaks training with no code change anywhere.
Agent Lifecycle
Awake() → base.Awake() registers the RPC communicator. Call it.
OnEpisodeBegin() → reset THIS agent's state
CollectObservations() → write the observation vector
OnActionReceived() → apply actions; this is your FixedUpdate
EndEpisode() → terminal; triggers OnEpisodeBegin on the next step
DecisionRequester.DecisionPeriod sets how many physics ticks pass between
decisions. With TakeActionsBetweenDecisions = true, OnActionReceived still fires
every tick with the last decision's actions — which is what makes force application
smooth instead of impulsive.
Multi-agent resets do not belong in OnEpisodeBegin
When N agents share one episode, N independent OnEpisodeBegin calls run in
arbitrary order and can half-reset shared state. Leave OnEpisodeBegin empty and let
one director reposition everything after ending the episode:
// Agent_FootballPlayer.cs — deliberately empty
public override void OnEpisodeBegin() { }
// Systems_EpisodeDirector owns the reset for all 22 players, together.
Observations
The observation vector is a contract. Change its size and every existing .onnx
fails to load — there is no migration, and --initialize-from will not help.
public override void CollectObservations(VectorSensor sensor)
{
Sensor_FootballState.Collect(sensor, _observationBuffer, /* ... */);
}
Rules that hold regardless of the game:
- Normalize to [−1, 1] yourself. Divide positions by a range constant
(
OBSERVATION_RANGE), speeds by a top speed. Then setnormalize: falsein the trainer YAML — running normalization on top of pre-normalized input fights self-play, because the input distribution shifts as the opponent improves. - Reuse a buffer.
CollectObservationsruns every decision for every agent.private readonly float[] _observationBuffer = new float[OBSERVATION_SIZE]; - Keep
OBSERVATION_SIZEa single const and setBrainParameters.VectorObservationSizefrom it in code. A scene-authored size that disagrees with what you emit is a silent shape mismatch. - Zero, don't omit, what an agent must not see. Defenders observe a zeroed play-call slice rather than a shorter vector — same shape, no leak.
Actions
// Only the brain that needs them gets the extra outputs.
behaviorParameters.BrainParameters.ActionSpec = _hasQuarterbackActions
? new ActionSpec(4, new[] { PLAY_CALL_SLOTS, 2 }) // 4 continuous + 2 discrete branches
: ActionSpec.MakeContinuous(2);
- Clamp continuous actions to [−1, 1] on arrival and scale to real units yourself. The policy is not obliged to stay in range.
- Do not hand a brain dimensions it must ignore. Dead outputs are dimensions the optimizer spends samples learning to zero.
- Apply in
FixedUpdateonly —OnActionReceivedis called from the physics loop. Never apply forces fromUpdate. Heuristicmust fill every segment. It is the manual-control and no-trainer path; a partial fill leaves stale actions.
Rewards
Split dense from terminal and keep both out of the Agent class:
Assets/Scripts/Reward/Reward_Progress.cs — per-tick, small, dense
Assets/Scripts/Reward/Reward_Terminal.cs — per-episode, large, sparse
AddRewardaccumulates;SetRewardoverwrites. PreferAddReward.- Dense rewards need a baseline reset. Rewarding a delta across an episode boundary pays out the jump from the old state to the new one. Seed the baseline on the first tick of a new episode and return without paying:
if (_lastSeenEpisode != _play.EpisodeIndex)
{
_lastSeenEpisode = _play.EpisodeIndex;
_previousBallY = ballY;
return;
}
- Give the do-nothing policy a cost. Without a per-decision time cost, standing still is a local optimum in any zero-sum game.
- Keep terminal magnitudes ~1.0 and dense ones ~0.01–0.05, so an episode's dense total cannot dwarf its terminal signal.
Self-Play
self_play:
save_steps: 50000
team_change: 250000 # 5x save_steps — the documented default ratio
swap_steps: 25000
window: 10
play_against_latest_model_ratio: 0.5
initial_elo: 1200.0
Set BehaviorParameters.TeamId per side. Both sides may share one behavior name or
use several — this project runs four (OffenseLine, OffenseSkill, DefenseLine,
DefenseCover) off one YAML anchor.
Judge on Self-play/ELO, never mean reward. Reward is zero-sum across the two
teams and hovers near 0 however strong the policies become. A flat reward curve in a
self-play run is the expected shape, not a failure.
Telemetry — instrument outcomes, not just reward
Mean reward cannot tell you how episodes ended. StatsRecorder averages within a
summary window, so writing 1 for the outcome that occurred and 0 for the others turns
each series into a rate:
StatsRecorder stats = Academy.Instance.StatsRecorder;
stats.Add("Play/TackleRate", outcome == Systems_PlayOutcome.Tackle ? 1f : 0f);
stats.Add("Play/TimeExpiredRate", outcome == Systems_PlayOutcome.TimeExpired ? 1f : 0f);
stats.Add("Play/NetYards", netYards);
Also record what the policy chose, not only what happened. A policy that collapses to one action is invisible in outcome rates alone — every rate still looks plausible while most of the action space goes unused.
Trainer Configuration
Config and run-id are paired 1:1 by name: Config/<Name><Phase><NN>.yaml ↔
--run-id=<name>_<phase><nn>.
behaviors:
OffenseLine: &football_brain
trainer_type: ppo
hyperparameters:
batch_size: 2048
buffer_size: 20480 # ~10x batch_size
learning_rate: 3.0e-4
beta: 5.0e-3 # entropy — keeps exploration alive in self-play
learning_rate_schedule: constant # a decay to 0 freezes the policy while
# opponents keep improving
network_settings:
normalize: false # observations are already in [-1, 1]
hidden_units: 512
num_layers: 2
time_horizon: 150 # long enough that terminal credit reaches
# back to the start of an episode
threaded: false # deterministic ordering
OffenseSkill:
<<: *football_brain # YAML anchors keep brains in lockstep
time_horizon shorter than an episode means terminal reward never reaches the
decisions that set it up. Size it against your episode length in decisions, not
ticks.
Running
.venv\Scripts\Activate.ps1
# In-editor smoke test: start the trainer FIRST, then press Play.
mlagents-learn Config\FootballBase03.yaml --run-id=football_base03
# Headless sweep — envs take CONSECUTIVE ports from --base-port.
mlagents-learn Config\FootballBase03.yaml --run-id=football_base03 `
--env=Builds\FootballEnv\PoFootball.exe --no-graphics `
--base-port=5010 --num-envs=6
tensorboard --logdir results
Operational rules that cost real runs when ignored:
- Overlapping port ranges hang the handshake with no error. Each env takes a
consecutive port from
--base-port. Two concurrent sweeps need disjoint ranges. - Record
--num-envsin the run'sMANIFEST.md. It changes how experience is batched, so two runs with the same YAML and different--num-envsare not comparable. - Kill TensorBoard before
--force. It holds Windows file handles and the wipe silently no-ops, leaving you resuming a run you thought you had cleared. - Shut down in order: trainer → envs → TensorBoard.
- 4–8 envs, leaving cores for torch.
time_scaleinengine_settingsspeeds the clock only; physics stays atdt = 0.02. It is not a dynamics change.
Promoting a brain
Assets/Agents/<Name>_v<NN>/
<name>.onnx
<Name>_Character.asset
MANIFEST.md — run-id, config, --num-envs, steps, final ELO
Overwrite .onnx in place to preserve the .meta GUID. Deleting and re-adding
regenerates the GUID and breaks every reference to it.
Failure Modes
| Symptom | Cause |
|---|---|
| Trainer exits at startup, port-ish error | Comms API mismatch between C# and Python |
| Handshake hangs forever | Overlapping --base-port ranges across sweeps |
.onnx refuses to load |
Observation size or ActionSpec changed since export |
| Reward flat near 0 in self-play | Expected — read Self-play/ELO instead |
| Reward great, behaviour bad | Reward is being farmed; instrument outcome rates |
| Agents freeze at episode start | Reset raced across N agents' OnEpisodeBegin |
| Agent ignores half its actions | A Rigidbody constraint is discarding them |
| Fatigue/load reads zero under load | Measured from the action vector, not applied force |
--force appears to do nothing |
TensorBoard is holding the results directory |
Related
- [[unitask]] — async without coroutines, for anything outside the physics loop
- [[vcontainer]] — injecting scene-instantiated Agents (see [[messagepipe]] for the event side)
- [[physics]] — the
FixedUpdateand fixed-Δt discipline actions depend on