dota-cv
Governs computer vision in Dota AI Coach — extracting information from the player's own screen via Windows Graphics Capture, ROI extraction, preprocessing, recognition, confidence scoring, temporal sm
SKILL.md.claude/skills/dota-cv/SKILL.md
GitHubDota Computer Vision
Allowed purpose
CV exists in this project to extract information already visible on
the player's own screen — nothing more. This is a narrower mandate than
"extract useful information about the game": if a fact isn't actually
rendered in the current frame, CV has no legitimate way to know it, full
stop. This mirrors dota-fairplay's VISIBLE_SCREEN classification
directly — CV is, definitionally, the system component that produces
VISIBLE_SCREEN-classified data, and only that.
Potential sources
All of the following are legitimate CV targets because they're rendered on the player's own screen during normal play — not because they're generically "visible" in some looser sense:
- draft portraits — hero picks/bans shown during the draft phase.
- top hero bar — the row of hero portraits/status shown during the match.
- selected unit HUD — stats/abilities/items for whatever unit is currently selected/inspected.
- visible inventory — item slots for a hero currently shown (the
player's own, or an enemy's when their portrait/HUD is on-screen
during a fight — see
references/allowed-sources.mdfor the exact boundary this implies). - scoreboard — the tab/score-panel view, when open.
- kill feed — the on-screen kill notification log.
- visible minimap state — only what the minimap actually renders (visible units/wards), never a fog-of-war-reconstructed view.
See references/allowed-sources.md for per-source detail on exactly what
"visible" means for each — several of these have a real, non-obvious
boundary (e.g. the top hero bar shows some info about heroes regardless
of current vision, which needs careful handling — see that file).
Architecture
Windows Graphics Capture
→ ROI (region-of-interest extraction)
→ preprocessing
→ recognition
→ confidence
→ temporal smoothing
→ ObservableState
Each stage's job, in brief (full detail in references/pipeline.md):
- Windows Graphics Capture — captures the actual rendered frame from the game window. This is the only legitimate input to the whole pipeline — everything downstream must trace back to an actual captured frame, never a substituted or cached "equivalent" value from another source (see the fair-play rule below).
- ROI — crops the captured frame down to the specific screen regions each recognition target needs (draft portraits, HUD, etc.), using known UI geometry.
- Preprocessing — normalizes a cropped region for recognition
(scaling, color normalization, etc.) — see
references/resolution-and-scaling.mdfor why this stage carries more weight in this project than in a typical CV pipeline. - Recognition — extracts the actual structured value from a preprocessed region (which hero portrait is this, what does this HUD text say).
- Confidence — every recognition result carries a confidence score; see the confidence-threshold rule below.
- Temporal smoothing — combines recognition results across frames to reduce jitter/noise before producing a stable observation.
- ObservableState — the CV pipeline's final output: structured,
confidence-scored observations, in the shape
dota-architecture's State Merger expects as CV's contribution toMatchState.
Full detail: references/pipeline.md.
Rules
- Process only visible information. Every recognition target must
correspond to something the player's own client actually renders —
see "Potential sources" above and
references/allowed-sources.md. - No hidden information inference. CV must never attempt to
reconstruct or estimate something not actually on screen (e.g. an
enemy's likely inventory based on gold/time heuristics rather than an
actually-visible portrait) — that's not a CV task at all, it's a
strategic inference task that belongs (if it belongs anywhere) in a
clearly-labeled, non-live-classified analysis layer, never disguised as
a CV observation. See
references/fairplay-coordination.md. - No screenshots stored by default. Captured frames/ROIs are
transient processing input, not a persisted artifact — see
references/privacy-and-storage.mdfor the narrow, explicit-opt-in exceptions (e.g. debugging a misrecognition) and how to handle them safely. - Prefer fixed-region classification over heavyweight detection when
geometry is known. Dota's UI has fixed, known layouts (the HUD,
portrait rows, scoreboard) — use that geometry directly (crop known
regions, classify a small fixed set of possibilities) rather than
reaching for general-purpose object detection where a much simpler,
faster, more reliable approach works. See
references/recognition-approach.mdfor when detection is actually warranted (e.g. minimap unit positions, which aren't fixed-region). - Account for resolution, aspect ratio, DPI, and UI scaling. Fixed-
region geometry is only fixed relative to the game's own UI
coordinate space — translating that to actual captured pixels requires
handling all four of these correctly, or ROI extraction silently
breaks on any player whose setup differs from whatever was used to
define the regions. See
references/resolution-and-scaling.md. - Require confidence thresholds. No recognition result reaches
ObservableState(or anything downstream) without clearing an explicit confidence threshold — seereferences/confidence-and-smoothing.mdfor how thresholds should be set per recognition target and how temporal smoothing interacts with them.
Coordinate with dota-fairplay
Every new CV feature — a new recognition target, a new source region —
is exactly the kind of change dota-fairplay requires classification
and documentation for (source/availability/visibility/
fairPlayClassification/justification) before it's considered done.
For CV specifically:
fairPlayClassificationshould beVISIBLE_SCREENfor essentially everything this skill covers — if a proposed CV feature seems to need a different (less restrictive) classification, that's a strong signal something has gone wrong in the design, not a reason to relax the classification.visibilitymust cite the actual screen element concretely (which region, under what condition it's rendered) — "the player can probably see this" is not sufficient perdota-fairplay's own guidance.- A FairPlay test (per
dota-testing) should assert the CV pipeline reports "unavailable," not a stale/substituted value, when the relevant region isn't actually visible in the current frame — seereferences/fairplay-coordination.mdanddota-testing's CV-testing guidance for exactly this "unavailable" case.
Relationship to other skills
dota-fairplay— the classification system CV output must satisfy; read together with this skill for any new CV feature, not after the fact.dota-architecture— CV is a sibling observation source to GSI, reconciled by the State Merger; "Computer Vision must not contain Dota strategy" is that skill's rule, restated and expanded here as "no hidden information inference."dota-gsi— the parallel observation pipeline; CV and GSI don't depend on each other, but both feed the same State Merger and both produceMatchState-bound observations.dota-testing— CV test requirements (known-capture test library, explicit "unavailable" case testing) are defined there; this skill defines what correct CV behavior is,dota-testingdefines how it's verified.