gamecore-trace-latency-diagnosis
Diagnose HttpCore endpoint latency in the gamecore repo from OTEL trace waterfalls — span attribution (CONNECT = Npgsql physical opens, Steam calls in SteamAuth), code call-site pinning, and the docs/
SKILL.mdskills/managed/gamecore-trace-latency-diagnosis/SKILL.md
GitHubDiagnosing HttpCore endpoint latency from trace waterfalls
Procedure proven on POST /user/login (~840 ms → root cause: sequential uncached Steam Web API calls + pool-minted DB connections). User supplies the trace waterfall + log lines from the analytics page.
1. Bucket the waterfall
- External HTTP GET/POST spans: match to log lines (sent/received timestamps confirm duration and single-attempt vs retry). These are usually the dominant cost.
- CONNECT spans (e.g.
CONNECT Handball-predev-origin): physical Postgres connection opens, emitted by Npgsql tracing wired inLibrary/GameCore.DAL/Databases/DatabaseContextPoolShared.cs(ConfigureQueryTracing). Multiple CONNECTs to the SAME database in one request = separate pools/contexts each opening cold — the AGENTS.md scoped-session anti-pattern. Sum them as "connection churn", distinct from query time. - SELECT/INSERT/UPDATE/DELETE spans: in this stack each is 0.4–4 ms; DB queries are essentially never the cause.
- Internal spans: match literal names to
StartActivity("...")calls. Nesting inflates parents (e.g.Authenticatinglooked heavy only because CONNECT opens sat inside it).
2. Attribute spans to code (known map, handball01 worktree)
Logging in→Container/HttpCore/Controllers/user/UserController.cs(~:509, insideLogin1_1, the POST /user/login endpoint).Authenticating→GameCore2/Services/Identity/UserService.cs(~:191,AuthenticateAsync).Lookup/Create User→UserService.cs(~:263).- Steam Web API calls →
GameCore2/Services/PlatformAuth/SteamAuth.cs:ValidateAsynccallsGetPublisherAppOwnership(:244) then:314) sequentially, uncached, per login.AuthenticateUserTicket(AuthenticateUserTicket's response carries only result/steamid/vacbanned/publishedbanned (no entitlements) — the ownership call cannot be replaced by it, only cached (per steamId+appId, TTL hours, existing primitiveCacheLogic.GetOrSetLayeredMessagePackCacheAsync) or parallelized (Task.WhenAll; watch the Soft-fail early-return that skips ticket validation for non-owners). - HttpClient names:
quickretry(Polly 1s/2s/4s waits — tail-latency risk only), defined inGameCore2/Extensions/HttpClientExtension.cs.
3. Gotchas
- Trace log categories can lie: login traces logged under
...dynasty.Centurion.CenturionControllerbut the endpoint isUserController.Login1_1. Match on route + log message text, not category. IsValidIpAsyncruns twice per login (UserController+UserService) → duplicateSELECT gc_configuration.ip_filters. Immaterial; note and move on.- Some internal spans (
SettingDapperRead) don't exist verbatim in source — DAL session bookkeeping; don't chase them.
4. Write the deliverable
docs/<endpoint>-latency-diagnosis.md with sections: Summary (bucket table with ms + % of total) / Trace evidence (span durations + log timestamps) / Root cause / Code call-sites (file:line table) / Ranked fixes (each with expected ms saved derived from the trace) / Observations.
- Pin EVERY code claim to file:line from actual reads — no invented citations.
- Diagnosis only unless asked:
git status --porcelainmust show only the new doc. - Median math: total − external-call time ≈ post-fix estimate; recommend p50/p95 over many traces for before/after.