Diagnostic method: how to debug a system you did not build

TL;DR: Establish impact, define the symptom precisely, bound the blast radius and the timeline, then form hypotheses you can eliminate rather than confirm. Mitigate before you root-cause. The order is the skill; the commands are lookup.

The symptom this explains

Two engineers are given the same failing service that neither has seen before. After twenty minutes one of them has narrowed the fault to a single dependency and has a mitigation running. The other has collected a great deal of output and can say that several things look unusual.

The difference is not which commands they knew. Both knew the commands. The difference is that every action the first one took removed a set of possible causes, and most of the actions the second one took added information without removing anything.

That is the property worth building a method around: an action is useful when it eliminates possibilities, not when it produces output. Each step below is chosen for how much of the search space it removes per minute spent.

The sequence

Establish impact. Who is affected and how badly? This decides urgency and whether you are allowed to take disruptive action. "All users, checkout failing" and "one internal dashboard is slow" deserve different responses, and starting here stops you over-reacting to the second.

Define the symptom precisely. Not "it is slow" but "p99 on the checkout endpoint went from 200ms to 3s at 19:40, affecting roughly 40 percent of requests". Precision bounds the search immediately: something that started at a moment has a trigger, and the affected fraction gives you a population to compare; a global setting can still affect only certain requests.

Bound it. Is it one region, one node, one customer, one endpoint? Every dimension you can rule out removes a branch. This is the highest-value step per unit of time and the one most often skipped.

Establish the timeline. What changed near the start? Deploys, config changes, feature flags, certificate rotations, traffic events, upstream incidents. Correlation is not proof, but it is where to look first, and most incidents do have a trigger.

Form hypotheses that are falsifiable. State the belief and the observation that would kill it. "If the new release's request path is the cause, a completed rollback with compatible state should restore latency" gives you a test with explicit conditions. Persistent data changes or an incomplete rollback can leave symptoms in place. "It might be the network" is neither.

Mitigate before root cause. If a rollback, a failover, a flag flip or shedding load restores service, do it and diagnose afterwards. The reason is not patience: while users are affected, every minute spent understanding the fault is a minute of impact you had the means to stop, and diagnosis is easier once the clock is not running. The cost is that some evidence is perishable, so capture what the mitigation will destroy before applying it, which usually means a heap dump, a copy of the logs, or a snapshot of the failing node.

Then root cause, validate, and prevent. Confirm recovery is real rather than coincidental, and turn the finding into a change with an owner.

Say the hypothesis and its falsifier out loud

State what you believe and what would disprove it, in words, before you go and look. This is not a presentation habit; it changes the quality of the diagnosis in two specific ways.

A hypothesis spoken with its falsifier can be checked by somebody else. "I think the new release changed the request path, and a completed rollback with compatible state should restore latency" invites a colleague to say that the rollback did not complete, or that the schema moved. The same belief held silently cannot be corrected by anyone.

It also forces the belief to be falsifiable at all. "It might be the network" survives being said aloud because nothing about it can be tested. Trying to attach an observation that would kill it is what exposes that it is not yet a hypothesis, and the attempt takes a few seconds.

Keep an evidence table during the incident

For a hypothetical checkout slowdown, record the hypothesis, test, result and remaining uncertainty in a few lines. “Only new pods fail” supports checking startup dependencies, but it does not prove a secret rotation. “The same digest works on old nodes” narrows the comparison to effective configuration and node conditions while leaving traffic differences open.

Choose a test that distinguishes competing explanations. If the failing request works through a pod port-forward, repeat it from the real caller's network context before clearing networking: port-forward bypasses part of that path. If a restart helps, record which state it replaced and what evidence was lost.

During handover, state the current user impact, confirmed observations and next bounded test. Keep proposed causes separate from established facts. Another responder should be able to continue the investigation without inheriting a guess as a conclusion.

Self-check

You are told "the service is slow". Before touching a terminal, name the questions you would ask and say what each one removes from the search space. Then answer a second one: a rollback has restored service and nobody yet knows why it was broken. Is the incident over? Work through both before reading on.

The questions are the first three steps, and what makes them worth asking first is what each eliminates. Asking who is affected and how badly separates "all users, checkout failing" from "one internal dashboard", which decides whether you are permitted to take disruptive action at all. Asking for the symptom precisely, as a metric, a magnitude and a start time, converts an adjective into a boundary: something that began at 19:40 has a trigger near 19:40, and a 40 percent affected cohort needs comparison by route and input, since a global configuration can still have selective effects. Asking how far it extends, by region, node, endpoint, tenant or client version, removes a whole dimension per answer. None of the three needs a terminal, and together they usually cut the candidate causes by more than any command would.

On the second question: user impact has ended, and the incident has not. The rollback is evidence rather than an explanation, and it is weaker evidence than it feels. It is consistent with the release being the cause, and also with a coincidental recovery, with the restart clearing an unrelated bad state, and with the real trigger being an upstream change that stopped at the same time. You do not yet know which, so the same fault can recur on the next deploy. What closes it is a cause that explains the specific symptom, a check that recovery is real rather than coincidental, and a change with an owner. Until then the correct description is mitigated, not resolved, and saying so is what stops the next release re-enacting it.

Next: start with a service map practises bounding a fault across boundaries, and the load test containment question runs the whole sequence under time pressure.

Apply the evidence-first approach to connection pool exhaustion, where a full application pool can coexist with an apparently idle database.