Google Incident Response & Production Debugging interview questions

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Tests whether you know that Kubernetes waits, and for how long. The default timings surprise most candidates, and they explain why a node failure hurts longer than people expect.

Both can be right, which is the whole point. Retransmissions mean a segment was not acknowledged, and the network is only one of the places that can happen.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

A two-second dependency should cost you latency, not three services. The mechanics are arithmetic: Little's law plus retry amplification. The fix is a budget, not a ban.

Repeated action items are a measurable defect with a mechanical cause. Answering 'culture' without a mechanism describes the symptom back rather than fixing it.

One disk metric explaining a whole cluster's weirdness is the point of this question. Restarting etcd first turns a slow cluster into a down one, because a restarted member has to catch up before it can vote.

Gradual drift produces no errors on the host that has it, only weirdness everywhere else. Sweep first, histogram second, repair third: the distribution shape names the culprit before you log into anything.

Slow leaks hide inside restart cycles and percentage thresholds. Plot weeks of absolute memory against deploys, prove it with profiler deltas, and alert on predicted time-to-limit instead.

Fork failures can reflect PID limits or memory allocation failure even when host RAM looks available. Find which ceiling broke, cap PIDs per cgroup so one team's leak costs one pod, and fix the unbounded spawn.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

A drain that hangs without erroring is almost always eviction being refused, and the two refusals that look identical from the outside have opposite fixes.

The disk is not full, the keyspace quota is. Defragmenting without disarming the alarm leaves the cluster read-only, and disarming without compacting puts it back within minutes.

The volume is in one availability zone and the scheduler cannot move it. Spare capacity in the other two zones is not spare capacity for this pod, and no amount of scaling helps.

The pod's resolver is configured to try five search domains before the real one. An uncached external hostname can trigger several search-expanded queries before its absolute lookup, and the failures are what you are paying for.

Every SDK fetches its credentials from a link-local metadata endpoint with a modest request budget. Enough containers refreshing at once and the endpoint starts refusing, which the SDK reports as an auth problem.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

An idle database with a starved application is the tell. The connections are held, not used. More database capacity does not release leaked connections.

Status codes describe an HTTP failure category. Locate the component that emitted the response, then use its logs to distinguish connection, capacity and application failures.

Three places, and which one depends on how the thing was started. Knowing which logs the runtime collects and which the application writes itself, and that journald may not persist across reboots, saves the first twenty minutes.

The gentlest scheduling question there is, and it still separates people who read the events from people who guess. Check whether the pod has a node before choosing between scheduling and startup diagnosis.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.