game-economy-ground-truth-audit
Use when asked to audit or sanity-check a game/fitness reward economy (XP, score, currency curves) across a whole catalogue, or after a fix makes a previously dead scoring path pay out. Also before ju
SKILL.mdskills/managed/game-economy-ground-truth-audit/SKILL.md
GitHubGround-truth economy audit
Auditing a reward economy by reading formulas produces plausible-sounding findings that are wrong about magnitude. Auditing it by reading the numbers the code actually pays produces findings with arithmetic attached, and it finds the outliers you would never have thought to look for.
Always generate ground truth FIRST, then fan out.
1. Dump what the code really pays
Write a throwaway test in the project's own test source set that walks the real catalogue through the real scoring functions and writes a TSV. Not a reimplementation of the formula — call the shipping code.
One row per catalogue entry, with the columns an audit needs to compare:
name group category metric weighted tier intensity sample score scorePerUnit secondaryScore
Rules that make the dump usable:
- Pick one realistic
sampleper metric and state it in the row (10 reps,60 s,30 min,5 km / 30 min,10 attempts). Comparisons are meaningless without a stated effort. - Emit a
scorePerUnitcolumn. Totals hide inversions; per-unit rates expose them instantly. - Append REFERENCE rows: what an ordinary session pays, what the biggest single award is (e.g. a skill claim), what one level costs. Every later judgement is a ratio against these, and without them a number is just a number.
- Mark columns that do not apply to some rows. A field defaulting to
2for entries that never route through it will be reported as a finding by every agent you dispatch unless you pre-empt it.
Then delete the test and the TSV at cleanup — it is scaffold, not coverage.
Traps: the --tests filter may silently match nothing even when the class
compiles; just run the whole suite. Check the file actually landed
(glob **/dump.tsv) — the working directory of a unit test is often the module
dir, not the repo root.
2. Model the extremes yourself before dispatching
Run the curve over realistic volumes, not the sampled one. The sample is for comparison; the extremes are where the economy breaks.
A flat per-unit rate is the classic defect: it is calibrated to one modality and silently applied to every other. Sweep the plausible range of each modality and convert the worst case into the currency the player feels — levels, not points:
def level(xp, cost=lambda l: 100*l):
lv = 1
while xp >= cost(lv): xp -= cost(lv); lv += 1
return lv
"One entry mints 9 levels" is a finding. "4000 XP" is a number.
3. Look down the stack before proposing a new table
Before anyone proposes a per-entry intensity or difficulty table, grep for one that already exists. Calorie/energy (MET) estimation, difficulty tiers or a skill tree, a progression graph, a rarity enum and analytics layers frequently already hold a calibrated value for exactly these entries, built for another purpose and never consulted by scoring. Reusing it is one rule instead of a second hand-maintained table that will drift.
Two parallel models of the same quantity in one package is itself the finding.
4. Fan out against the numbers
Dispatch read-only scouts with the TSV path in shared context and an explicit instruction: use these numbers, do not recompute from the formulas, do not speculate. Useful axes, one per agent:
- the newly-paid or least-reviewed curve family
- the competitive score (a leaderboard) as distinct from the personal one
- descriptive correctness per entry (group, category, metric, flags)
- the progression/claim economy against session rewards
- achievement/title reachability against the real catalogue size
Require every finding to be tagged DEFECT (contradicts the code's own stated
rule) or BALANCE (a judgement the owner owns), and to cite path:line or a
TSV row. List known-and-deliberate decisions in the brief so they come back
under ## NOT FINDINGS instead of as findings.
5. What this reliably catches
- Modality-blind flat rates — one unit priced identically across activities whose real cost differs 10x.
- A bonus window calibrated to one member of the family, unreachable by the others and automatic for one of them.
- A missing taper on a competitive SUM. If the personal currency damps volume and the ranked one does not, the cheapest movement in the catalogue is the dominant strategy and it compounds without bound. Work the exploit with units-per-minute as the cost proxy.
- Achievement thresholds above the catalogue ceiling — unearnable content padding a denominator forever. Compare every threshold to the real count.
- Invariants enforced only in the UI. Grep the gate function's callers: if the only ones are screens, the data layer will mint the reward out of order.
- Wholesale category mapping where a progression axis is mistaken for a taxonomy axis, so mixed members land in the wrong bucket.
6. Do not rewrite the economy unasked
An audit's deliverable is the audit. Separate the clean defects (no numbers anybody earned against change) from the balance calls, and present the balance calls as costed options with the arithmetic. Check whether a recompute path already exists before repeating a "changing this would restate history" argument — if the code can already rescore from stored inputs, that objection may be obsolete, and the comment asserting it is part of what you are auditing.
Acting on the findings once the owner chooses is score-economy-rework-safety.