Benchmarking a rendering change (A/B in one run)

The fastest reliable method: temporarily add a global toggle for the measured thing, flip it automatically every few seconds inside ONE game run, and log running averages of GPU frame time (GL timer query) and FPS for both variants. Then remove the temporary code and rebuild.

One run with alternating phases cancels out scene, thermal and clock differences between separate runs.

Facts that save time

• src/view/frame_profiler.h measures only CPU time and has no long averages - useless for GPU passes. The rendering script only enqueues commands; the GPU cost of an FBO clear or fullscreen quad never shows there.
• FPS alone hides GPU costs: on the dev machine (RX 5700, ~1080p) the game is CPU-bound - ~1110 FPS with a ~0.5 ms GPU frame. Always report the GPU time, not just FPS.
• Always pass /range to launch straight into the shooting range, no input needed: ../build/current/Hypersomnia /range. It's a URL-style location argument (src/cmd_line_params.h, also /menu, /editor, /tutorial). Confirm in the log: Activity SHOOTING_RANGE was specified from the command line. and Launched mode: SHOOTING_RANGE. Without it, the game launches last_activity from hypersomnia/user/runtime_prefs.json - which is whatever the user did last (editor, menu, a server...) - and only if skip_tutorial is true, else the tutorial. launch_at_startup: MAIN_MENU in the config would send it to the menu too.
• A different scene (editor, a map on a server) needs a different measuring setup - ask the user if the range isn't representative.
• A specific map, with a fixed camera: open it in the editor with --edit <project dir>, e.g. ../build/current/Hypersomnia --edit user/projects/de_cyberaqua_bench (cwd = hypersomnia/). Confirm Launched mode: EDITOR and that no DTOR finished: ~editor_setup follows right after loading - builds before 2026-10-01 fell through to the project selector with --edit (missing break in work.cpp).
  • Never benchmark on the official content/arenas/* or the user's own user/projects/*: the editor writes navmesh.nav, editor_view.json and autosaves there. user/projects is a symlink to ../content/arenas, so a copy there lands in the repo too. Copy the arena to the scratchpad instead - --edit takes an absolute path - rename its json to match the folder, drop last_saved.json, and set meta.name.
  • Builds before 2026-10-05: --edit opened the editor only when the last activity was the editor, and overwrote user/editor_last_project.txt with the scratch path - the user then edited the scratch copy unknowingly. Since then --edit always opens the editor and leaves both untouched; --playtest starts playtesting right away. Still never name the copy like the original.
  • The editor camera comes from editor_view.json (panned_camera.transform.pos, zoom) - set it to frame what is measured, e.g. zoom 0.45 at the average position of the lights to catch many of them.
  • A frame "saved" by the user reaches editor_view.json only when their editor quits - ask them to quit it first, and check that the camera lies within the map (the min/max of the nodes' pos in the map json). Num visible entities: 0 and ~1200 FPS in the profile mean an empty camera, not a fast scene.
  • Downloaded maps: hypersomnia/user/downloads/arenas may be a symlink (to ~/poprawionemapy in Oct 2026) - resolve it with readlink -f before deciding which copy the user edits. ~/.config/Hypersomnia/... holds older copies.
  • Editor playtests run the map's playtesting.mode. quick_test spawns no bots even with spawn_bots - set "mode": "bomb_defusal" in the scratch copy to measure bots.
• Screenshots of the game: import -window $(xwininfo -name <title> | awk '/Window id/{print $4}') out.png. Never import -window root - it captures whatever the user has on screen. A window on another i3 workspace is unmapped and comes out black; on X11 without a compositor it still renders (the timings stay valid).
• Never use --apply-config for a benchmark variant: the patch lands in user/runtime_prefs.json when the game quits (even on timeout's SIGTERM), silently changing the user's settings. Back up runtime_prefs.json and diff it after the run, or toggle the setting in code with a /* BENCH TEMP */ flag. The editor also saves its camera to editor_view.json on quit - reset it before every run so the scene stays the same.
• CPU-side light work: the profiler has light_ray_casts, light_aabb_queries, lights_recalculated, lights_reused and light_jobs_cpu_us per frame (frame_profiler.h, summed by the light jobs in launch_visibility_jobs.cpp from the per-thread thread_physics_query_counters). Log them in the A/B harness instead of the GPU query when the measured thing is CPU work. The light jobs run in parallel on the thread pool, so their summed CPU time barely moves the FPS on a many-core machine - report the CPU sum, not just FPS.
• Uncapped already: vsync_mode is OFF in default_config.json, and max_fps is disabled (OFF_max_fps in runtime prefs). Verify if the numbers look capped (e.g. exactly 60/144/400 FPS).
• Launch with cwd = hypersomnia/: cd hypersomnia && timeout 76 ../build/current/Hypersomnia /range > <scratchpad>/bench.txt 2>&1. From the repo root the game fails to read its config and litters the root with empty cache/, user/, logs/.
• LOG output goes to stdout, so it lands in the redirected file even when timeout kills the game.
• The user allows launching and killing the game by yourself for benchmarks.
• Check that no other game instance runs before measuring: ps -eo pid,pcpu,lstart,args | grep -i "[h]ypersomnia". The user often keeps one open (launched via ninja run -C build/current), or plays the release AppImage between runs - it runs as /tmp/.mount_*/usr/bin/Hypersomnia, which a build/ grep misses, and took 320% CPU in one case. It skews the absolute CPU numbers and FPS - ask them to close it rather than killing it, then rerun. An A/B in one run keeps the ratio roughly right even then.
• Detect disturbances during every run, not only before it: the user compiles, plays and runs other agents meanwhile. Sample every second with top -b -n2 -d1 (the second iteration - ps pcpu is a lifetime average and misleads) for ninja|clang++|cc1plus|ld.lld|Hypersomnia|Xvfb above 3% CPU, and drop and rerun a run that had any. Also drop runs whose BENCHPROF shows Debug details (0.4-0.5 ms) - the user opened the performance overlay in the window. A game started from Claude's shell inherits nice 5 (only root can lower it) - fine when the machine is idle.
• Name the benchmark binary: a symlink in build/current (e.g. HypersomniaBenchJedrula -> Hypersomnia) makes the process distinct from the user's game. Kill it with pkill -x HypersomniaBenc (comm is cut to 15 chars) - pkill -f <pattern> kills Claude's own shell, whose command line contains the pattern.
• A separate worktree for the instrumentation (git worktree add --detach, submodules copied as described below): the user and other agents keep building the main checkout. git stash in a worktree shares the stash list with the main checkout - save the instrumentation with git diff > patch instead, and git apply --3way it onto a newer master. Port the final fix to the main checkout without the /* BENCH TEMP */ hunks and commit only those files (git commit -o).
• Ready instrumentation in instrumentation/ next to this file (header-only, include them where marked):
  • bench_gpu_segments.h - the per pass GPU timing below. Hooks: bench_gpu::set_fbo(id) in fbo::set_as_current_impl (0 in set_current_to_none_impl), set_prog(id) in shader_program::set_as_current_impl, set_tag(int) for the light_pass uniform in shader_program::perform(set_uniform_command), on_draw(tris) after glDrawArrays, begin_frame() / end_frame(frame_s) around the renderer_backend.perform loop in work.cpp. BENCH_GPU=0 switches the blocking readback off and keeps only BENCHFRAME (FPS, p50/p90/p99/max frame time).
  • bench_jobs.h - CPU time of the thread pool jobs per frame: bench_jobs::scope(bench_jobs::FOW) at the top of fow_job, LIGHT in light_job (launch_visibility_jobs.cpp), ILLUM / GAME_GUI / POST_GUI in the jobs of work.cpp, HELP_WALL around help_until_no_tasks + wait_for_all_tasks_to_complete, end_frame() after it. The other slots served as plain counters (ns[slot] += n, printed /1e6 per frame).
  • bench_cpu_timers.h - per simulation step: report_and_reset() + scope(STEP) at the top of standard_solve, scope(...) in the AI and pathfinding functions. When report_and_reset() returns true, log the cosmos profiler too (performance.prepare_summary_info(); performance.summary(s); - needs augs/templates/introspect.h in that file).
• "Help tasks" in the game thread profile is the wait for the thread pool - the frame waits for its longest job. Time each job (bench_jobs.h): the parallel light jobs sum up high but rarely matter; a single long job does.
• Which pass eats the GPU: time segments of the command stream - glQueryCounter(GL_TIMESTAMP) in renderer_backend::perform at every change of the bound FBO (fbo::set_as_current_impl) or program (shader_program::set_as_current_impl), read back blocking at the frame end, summed per (fbo id, program id). Log the ids at creation (fbo::fbo, the path ctor of shader_program) to name them; all_necessary_fbos::apply creates them in a fixed order. Split further by a tag: the light_pass uniform (hook shader_program::perform(set_uniform_command)) or the dedicated_buffer of a drawcall_dedicated_command. Trust the total and big segments only - the cost of an FBO switch can land in a neighbouring segment. The blocking readback lowers the FPS: measure FPS in separate runs without it.
• GLSL: return early instead of skipping work in an else: on radeonsi a uniform if (mode == 2) { cheap } else { expensive } ran at the cost of the expensive branch (0.90 ms); returning right after the cheap path got 0.29 ms.

Frame pacing / smoothness (judder), and comparing two builds

FPS and GPU averages say nothing about judder. Use smoothness/ next to this file:

• smoothness.patch (git apply it; applies to HEAD and to tag 3.0.0) adds src/bench_smoothness.h and hooks in work.cpp. With HYPERSOMNIA_SMOOTHNESS_BENCH=<out.csv> in the editor it starts playtesting after 3 s (like Space), holds A+W from 6 to 10 s, dumps one CSV row per frame and quits by SIGTERM. Per frame: game frame start and delta, GPU timestamps after the frame's commands (GL_TIMESTAMP, so fill rate counts), swap times, the drawn camera position (after pixel snapping), the raw camera, the viewed character, logic steps. Not for committing: GL_TIMESTAMP does not exist in WebGL.
• analyze_smoothness.py name=run.csv name=run2.csv ... groups runs by name and prints p50/p99/p99.9/max of: display intervals, GPU and swap times, latency, position judder (deviation from a moving average, in screen px), the step per simulated 144 Hz refresh, backward camera steps per second, and hitches. The display time of a frame = max(GPU end, swap end) - right for X11 without a compositor, vsync off.
• Run 3-5 times per build, alternating builds. The bench script needs a window in view; a run that logs Launched mode: SHOOTING_RANGE or WM_DELETE_WINDOW was disturbed by the user - drop it and rerun.
• Use the same scratchpad copy of the map for both builds (3.0.0 loads HEAD's map json fine) and restore its editor_view.json before every run.

Another build (e.g. a release tag): git worktree add /home/pbc/Hypersomnia_<tag> <tag>, then copy the submodule directories from the main checkout instead of cloning (git ls-tree -r <tag> | awk '$2=="commit"{print $4}'; compare with HEAD first) and delete the .git files inside the copies, or every git command in the worktree fails. Build with BUILD_FOLDER_SUFFIX=fast cmake/build.sh RelWithDebInfo x64 -DGENERATE_DEBUG_INFORMATION=0, then ninja in build/current (~15 min, run it in the background). Copy hypersomnia/user/runtime_prefs.json into the worktree. Before commit a5c684fd5 --edit falls through to the project selector: add the break; after launch_editor(params.editor_target);. Remove the worktree when done: git worktree remove --force <path> (the build inside is the only thing lost).

What the smoothness numbers showed (Oct 2026): the judder came from the camera logic, not from frame times - backward 1 px camera steps from pixel snapping and 1 px "snap to target" jumps of the smoothing (fixed in 39f1b07fd). Frame times were steady on both HEAD and 3.0.0; rare 4-5 ms stalls inside renderer_backend.perform (0-1 per 4 s, random moments, both builds) are driver noise. Deterministic stalls happen only at loading: texImage2D uploads (8-20 ms) when the editor opens, and the first LIGHT_SHADOW_MASKS upload (7 ms) when playtesting starts.

Gotchas: FPS caps and vsync

Check these before trusting the numbers. A capped frame rate makes both variants look identical. Suspicious FPS values are the round ones: 60, 144, 165, 240, 400, or anything exactly flat across phases.

• In-game vsync: window.vsync_mode (OFF / ON / ADAPTIVE, src/augs/window_framework/window_settings.h). Default OFF in hypersomnia/default_config.json, but hypersomnia/user/runtime_prefs.json and any file in hypersomnia/user/conf.d/ override it. Grep all three for vsync_mode.
• In-game FPS limit: window.max_fps is a maybe<int>. In JSON, "max_fps": 400 means ENABLED at 400, "OFF_max_fps": 400 means disabled (value kept). Values below 10 are ignored (work.cpp, max_fps.value >= 10). The limiter method max_fps_method (SLEEP / SLEEP_ZERO / YIELD / BUSY) changes the frame pacing, but not the GPU time.
• Driver vsync (Mesa, AMD): the driver can force vsync regardless of the game - vblank_mode env var or ~/.drirc. Launch with vblank_mode=0 to be sure: vblank_mode=0 timeout 76 ../build/current/Hypersomnia /range ....
• Compositor / hidden window: a minimized or fully covered window may get throttled by the compositor or the driver. The game itself does not throttle unfocused windows. Keep the game window visible during the run.
• Swap moment: performance.swap_window_buffers_when (AFTER_HELPING_LOGIC_THREAD by default) moves where the frame blocks. Keep it the same for both variants. Since the toggle lives in one run, this is automatic.
• Settings changed in the game menu land in runtime_prefs.json on a graceful quit. timeout kills with SIGTERM, so a benchmark run shouldn't change them - but don't open the settings mid-benchmark.
• The GPU time is immune to all of the above. GL_TIME_ELAPSED measures the executed commands only, so even with a cap it gives the right diff. Only the FPS column gets flattened by caps.

Steps

1. Toggle. Add a global flag next to the measured code, e.g. in the rendering script's header:

   /* BENCH TEMP */
   inline std::atomic<bool> bench_skip_<thing> = false;
   

   Read it once per frame (const bool bench_skip = bench_skip_<thing>.load();) and skip ALL of the measured work with it: FBO binds and clears, draw calls, and the composite fullscreen quad. Mark every temporary edit with /* BENCH TEMP */.

2. Measure. Paste bench_snippet.cpp (next to this file) into src/work.cpp, around the renderer_backend.perform(...) loop in the render thread (search for rendering_result.clear();), and add #include "augs/graphics/OpenGL_includes.h" after the renderer_backend.h include. Set the flag name in the snippet. It:

   • waits 8 s of warmup, then alternates 3 s phases: even = WITH the thing, odd = WITHOUT,
   • drops the first 0.5 s of each phase (commands are built a frame or two before they run on the GPU),
   • wraps each frame's GL commands in a GL_TIME_ELAPSED query (ring of 8, read back 8 frames later),
   • logs cumulative averages after every WITH+WITHOUT cycle as BENCH cycles=....

3. Build with the build MCP tool.

4. Run ~76 s (8 s warmup + ~11 cycles), then grep:

   cd hypersomnia && (vblank_mode=0 timeout 76 ../build/current/Hypersomnia /range > <scratchpad>/bench.txt 2>&1); grep -E "BENCH|GL error|Launched mode" <scratchpad>/bench.txt
   

   Wait until the diff stabilizes over the last few cycles. The first cycle is noisy.

5. Revert only the temporary edits: git restore the touched files, but only after checking with git diff that they contain no other uncommitted changes of the user. Otherwise remove the /* BENCH TEMP */ hunks by hand.

6. Rebuild. The binary still contains the toggling code until you do. Don't skip this.

7. Report a table: GPU frame time and FPS with/without, the diff in ms and as % of the GPU frame, sample counts, resolution (window size from runtime_prefs.json) and GPU (lspci | grep -i vga). Scale the estimate for other resolutions by the pixel count, and state clearly that other hardware wasn't measured. Mention the memory cost of any removed FBO: width × height × 4 B for RGBA8 (+ stencil if WITH_STENCIL).

Reference results

HEAD (shadows on) vs 3.0.0, walking on de_cyberaqua in an editor playtest (Oct 2026, RX 5700, 1918x1060, zoom 0.8, X11/i3 without a compositor, 5 runs each): 769 vs 866 FPS, GPU 1.27 vs 0.60 ms per frame (HEAD is GPU-bound), latency from the frame start to the display 4.6 vs 3.1 ms - the GPU-bound frames queue up. Judder identical before the camera fix; after it, the per-refresh step deviation went 0.51 -> 0.17 px (p50).

The light shadows cache on de_cyberaqua in the editor (Oct 2026, RX 5700, 25 lights in view, all with heights, zoom 0.45): light jobs 3.06 → 0.50 ms of CPU per frame summed over threads (6x less), AABB queries 1319 → 25, 0 ray casts either way (lights with a height skip the visibility polygon and use only AABB queries), 407 → 467 FPS. With point light heights off (the visibility polygon path, 19 lights at another camera): 2471 → ~1 ray casts per frame, 3.96 → 0.18 ms of CPU, 409 → 517 FPS. GPU time +0.07 ms with the cache although it draws the same geometry - likely GPU clocks at the higher FPS, not a cost.

The sun shadow guard band (384 px past two screen edges) on the same scene: +0.006-0.010 ms GPU on a 1.9 ms frame (~0.5%), FPS unchanged, +5 MB of VRAM at 1080p.

de_facing_worlds regression (Oct 2026, editor playtest standing still, 24 lights in view, 1918x1060): 3.0.0 895 FPS / GPU 1.01 ms, HEAD 360 FPS / GPU 2.70 ms. Per pass: GROUND in illuminated.fsh 0.29 -> 0.90 ms (the layers of the ground overlap many times and every fragment computed posterize, hue and sun shadows - up to 18 fetches), point light masks 0.07 -> ~1.2 ms (passes 2-4, 7, removed and hue light FBOs). Fixed by resolving the light once per pixel (light_resolve, GROUND 0.29 ms + 0.13 ms resolve) and drawing the hue light only when some light doesn't keep its hue through walls: ~480 FPS / GPU ~2.0 ms. Skipping pass 3 with a scissored reach saved only ~0.05 ms - not worth the mask invariant.

The shell smoke overlay_smoke FBO pass with no shells on screen (Sep 2026, RX 5700, 1918×1060): 0.518 vs 0.498 ms GPU per frame (+0.02 ms, 4%), no FPS difference (1110 both), 8 MB of VRAM.

de_jedrula (Oct 2026, RX 5700, 1918x1060, a big map of 64 px wall tiles) - the slowdowns were CPU, not lights:

• Editor with the navmesh overlay on drew every cell of the map (~26k quads): GUI 1.6-2.7 ms of CPU, ~200 vs ~370 FPS off. Fixed by drawing only the cells in view (34406820e).
• Playtest by the portal ~425 FPS with only 2 ms of GPU: the thread pool waited 1.65 ms for fow_job - the viewer's fog of war polygon, single-threaded, ~500 rays over the whole fog of war area (size * 2 = 5100x2900 px). Rays only within the screen and the field of view angle (only_within of the visibility request, 172c5650b): 0.29 ms, ~835 FPS. Checked against the full polygon on a grid of 8 px cells within the screen and the angle - at most 2 of 58140 cells differ; the eye must stay off the bounds of the rect (looking straight up broke the polygon).
• Tried and dropped: the fog of war as shadows extruded from every obstacle (like the masks of lights with a height, no ray casts). CPU 0.75 ms for the full area, 0.27 ms clipped, but the long overlapping shadows of the many tiles overdraw the stencil on screen: +0.07-0.09 ms GPU (~2300 triangles instead of ~100), and ~790 vs ~830 FPS against the clipped ray casts. Rasterizing off screen costs nothing - the overdraw on screen does.
• Bots (9, HARD, bomb_defusal): the AI 0.1-0.3 ms per step, pathfinding spikes up to ~1.8 ms in a step (find_path_across_islands_many), 630-1130 FPS - not what dropped the FPS.