Debugging Methodology (Verbose)

Core Patterns

The Diagnostic Loop

Debugging is the scientific method under time pressure. The loop is: observe a fact, form a hypothesis that explains it, design the cheapest experiment that can disprove the hypothesis, run it, update.

The step engineers skip is designing the experiment. They jump from hypothesis straight to a fix, and a fix that appears to work confirms nothing — it may have perturbed timing, masked the symptom, or been irrelevant to a bug that is intermittent anyway.

Keep a running log. It costs seconds and stops you re-testing the same theory at hour three:

FACT   17:02  500s began 16:40, exactly at deploy of a3f19c2.
FACT   17:05  Only /checkout. /cart and /orders are clean.
HYP-1  Serialiser regression in a3f19c2.
TEST   Revert a3f19c2 in staging, replay the failing payload.
RESULT Still fails. HYP-1 DEAD.
FACT   17:20  Staging fails too — so it is not traffic volume.
HYP-2  Data-shape dependent: fails only for carts with a null variant.
TEST   Construct one cart with a null variant, one without.
RESULT Null variant reproduces at 100%. HYP-2 LIVE.

Two properties make an experiment good: it is discriminating (its two possible outcomes point at different causes) and it is cheap (seconds, not a deploy). Prefer a query over a deploy, a local call over a query, a unit test over all of them.

Binary Search Over Space

A request crosses many components. Rather than tracing it end to end, test the midpoint and discard half the system.

client → CDN → load balancer → API → cache → database
                                ↑
                        check here first

If the API access log shows the request with a 200, everything to its left is fine and the problem is downstream. One observation, half the system eliminated. Repeat: about three checks to isolate one component in an eight-hop path.

The same trick works inside a process — bisect by removal:

# 30 middlewares, one is dropping the auth header. Disable half.
MIDDLEWARE_DISABLE="$(sed -n '1,15p' middleware.list | paste -sd,)" ./run-repro.sh

If the bug persists, it is in the enabled half; if it vanishes, in the disabled half. Feature flags, plugin lists, and config blocks all bisect the same way.

Binary Search Over Time

When the code used to work, the failing change is findable in logarithmic time.

git bisect start
git bisect bad HEAD              # known broken
git bisect good v2.14.0          # known working
# git checks out the midpoint. Test it, then report:
git bisect good                  # this revision is fine
git bisect bad                   # this revision is broken
git bisect skip                  # unbuildable — exclude it
git bisect reset                 # done; return to your branch

Automate the verdict and walk away. Any command whose exit status is 0 for good and non-zero for bad will do:

git bisect run pytest -x tests/test_checkout.py::test_null_variant
git bisect run ./scripts/repro.sh          # exit non-zero when it reproduces

Two cautions. First, have a custom script exit 125 to mean "cannot test this revision" — bisect treats that as a skip rather than a verdict. Second, if the last good revision is weeks back, bisect over merge commits first (git bisect start --first-parent) to find the offending merge, then bisect inside that branch.

If the bug is likely in one file's history, narrow the candidates before you bisect at all: git log -S'connection_pool' finds commits that changed the number of occurrences of a string, and git log -L :price_cart:engine.py shows the history of a single function.

"It Works on My Machine"

Treat this as a successful experiment, not an excuse. You have two systems, one passing and one failing, and the bug must live in their difference. The work is to enumerate and shrink that difference:

Axis	How to compare
Runtime version	python -V, node -v on both
Dependency graph	Diff the lockfile, not the manifest
Environment	env | sort on both, then diff
Data	Row counts, nulls, encodings, one real failing record
Config	Diff the rendered config, not the templates
Clock / locale	Timezone, LANG, DST edges
Network	Egress rules, proxies, DNS resolution
Build	Debug vs release, optimisation level

Then bisect the diff: move one axis of the working machine toward the broken one until it breaks. That axis is the cause. Containers help by shrinking the diff in advance, but they do not eliminate it — the host kernel, mounted data, and injected secrets still differ.

Reading a Stack Trace

Three questions, three places to look.

Traceback (most recent call last):
  File "app/api/checkout.py", line 88, in post          <- your code: the entry point
    total = price_cart(cart)
  File "app/pricing/engine.py", line 41, in price_cart
    return sum(line_total(l) for l in cart.lines)
  File "app/pricing/engine.py", line 52, in line_total
    return item.variant.price * item.qty               <- your code: deepest frame you own
  File "site-packages/orm/proxy.py", line 210, in __getattr__
    raise AttributeError(...)                          <- library frame: where it surfaced
AttributeError: 'NoneType' object has no attribute 'price'

• What broke? The innermost frame and the exception message. Here: something is None where a variant was expected.
• Where did the bad value come from? The deepest frame in your code — engine.py:52. Library frames below it are usually just the messenger.
• Why was it called? The outermost of your frames — the entry point and its inputs, which you need in order to reproduce.

For chained exceptions, the block after The above exception was the direct cause of (or Java's Caused by:) holds the original failure; the outer layers are wrapping. Read to the first cause.

Async and threaded traces are truncated at the scheduling boundary — the frames that queued the work are absent. Propagate a request id through a context variable, or you will get a trace that starts at the event loop and tells you nothing about who asked.

Minimal Reproduction

A bug you can reproduce in one file is a bug you have nearly explained. Shrink aggressively:

1. Fix determinism first — pin the seed, freeze the clock, use one fixed input record. An intermittent repro cannot be shrunk, because you cannot tell "removed the cause" from "got lucky".
2. Cut breadth: one request instead of load, one row instead of the table, one service instead of the mesh.
3. Cut depth: replace the framework with a direct call, the database with a literal, the queue with a function call.
4. After each cut, re-verify it still fails. If it stops failing, the thing you just removed is implicated — put it back and keep it.

This is delta debugging done by hand, and tools automate it (creduce for C, property-test shrinkers such as Hypothesis for the input space). The shrunken case becomes the regression test, so the effort is never wasted.

Choosing Your Instrument

Situation	Reach for	Why
Fast local repro, tangled state	Interactive debugger	Inspect everything, step, mutate, retry
Production, cannot pause	Structured logging	No stop-the-world; queryable after the fact
Latency or causality across services	Distributed tracing	Spans show where the time and the failure went
Intermittent, low frequency	Sampling profiler / always-on tracing	Catches the rare event without a repro
Hang or deadlock	Thread/stack dump of the live process	Shows what every thread is blocked on
Hard crash	Core dump + post-mortem debugger	The only surviving state
"Something changed"	Metrics diff around the deploy	Cheapest first look, always

Conditional breakpoints beat stepping when the failure is on iteration 40,000: break on cart.id == 88213 rather than holding down continue. Log at the boundaries you cannot step across — process edges, network calls, queue handoffs — and carry a correlation id so lines from different processes can be joined.

Heisenbugs

A heisenbug changes or disappears when you observe it. The mechanism is real, not mystical: a log statement adds I/O and a lock, allocation shifts memory layout, a debugger halts one thread while others run on, and a debug build disables the optimisation that exposed the race.

If instrumentation makes the bug vanish, you have learned something concrete — it is timing- or layout-sensitive, which points at data races, unsynchronised shared state, uninitialised memory, or use-after-free. Adapt the technique:

• Observe from outside the process — tracing, strace, packet capture, kernel counters — so you do not perturb the window.
• Record cheaply inside a hot path: a fixed-size ring buffer written without locks, dumped only on failure.
• Use tools built for the class: thread and address sanitizers, race detectors, deterministic record/replay.
• Increase pressure rather than visibility: more concurrency, slower disks, injected latency, fewer cores. Make the race more likely, then catch it.

Never "fix" a heisenbug with the sleep or the log line that made it go away. That converts a reproducible failure into a rare one that will return in production at a worse time.

Handing Off

Live diagnosis ends when the defect is understood and mitigated. The systemic question — why the class of bug was possible, and why detection took as long as it did — belongs to the postmortem process (root-cause-five-whys), and the minute-by-minute record belongs in incident-timeline-creation. Do not try to run all three at once during an outage; capture facts with timestamps as you go so the other two are cheap afterwards.

Common Anti-Patterns

❌ Changing code to see what happens. Shotgun edits produce a system that works for unknown reasons and will break again. ✅ State a hypothesis and the expected result before editing anything.

❌ Making several changes at once. When it passes, no single change is credited, and you carry three unexplained edits forward. ✅ One variable per experiment.

❌ Reading only the exception's top line. The message says what surfaced, not what caused it. ✅ Find the deepest frame you own and the first "caused by".

❌ Debugging where the symptom appeared. The trace shows where the bad value exploded, not where it was created. ✅ Trace the value backwards to its origin.

❌ Dismissing "works on my machine". It is a passing control in an experiment you have already run. ✅ Enumerate and bisect the environment difference.

❌ Never shrinking the reproduction. A ten-minute end-to-end repro caps how many hypotheses you can test per hour. ✅ Spend twenty minutes making it a ten-second test; you will get it back.

❌ Adding sleep or retry until the failure stops. The race is still there, now rarer and harder to catch. ✅ Fix the synchronisation or the ordering assumption.

❌ Bisecting by hand across hundreds of commits. ✅ git bisect run with a scripted verdict.

❌ Trusting memory over a written log. By hour two you will re-test a dead theory and argue about whether you already tried it. ✅ Keep the fact/hypothesis/result log in the incident channel.

❌ Stopping at the first plausible explanation. Plausible is not verified. ✅ Confirm by prediction: the theory must say what happens when you change X, and it must be right.

❌ Ignoring a fact that does not fit. The one unexplained observation is usually the whole bug. ✅ A correct theory accounts for every fact, including the inconvenient one.

Debugging Checklist

• Symptom stated precisely — what, when it started, for whom, how often
• Reproduction exists and is deterministic
• Reproduction shrunk toward a single file / single input
• Current hypothesis written down and falsifiable
• Next experiment is the cheapest one that splits the space
• One variable changed per experiment
• Fact/hypothesis/result log kept as you go
• Stack trace read to the deepest owned frame and the first cause
• Recent changes checked; git bisect run used if it is a regression
• Environment diff enumerated when only some machines fail
• Instrument chosen deliberately: debugger, logs, tracing, or dump
• Timing sensitivity considered before adding in-process logging
• Explanation accounts for every observed fact, not just the loudest
• Fix verified by prediction, not only by the symptom going away
• Shrunken reproduction committed as a regression test
• Systemic cause handed to the postmortem process, not fixed and forgotten