Stripe Incident Response & Production Debugging interview questions

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Load balancing makes a per-host fault look like a flaky library. The exact rejection string tells you which way the offending clock leans before you log into anything.

The leaf is valid and the root is trusted, yet verification fails. The gap lives in the middle of the chain, and the exact verify code names it before you open a single config file.

The outage window closed, but every process born since then still tries to fetch secrets at boot with nothing to fall back on. Why restarts keep reopening an incident that already ended.

Bandwidth graphs look healthy and the partner swears they are fine. The wall is ports, not bits: the available source-port budget depends on the gateway and its address allocation.

Wall-clock schedules meet daylight-saving transitions: spring forward deletes an hour of cron targets, fall back repeats one. Schedule in UTC, add idempotency keys, rehearse in staging.

A loop with no step cap has no natural ceiling, and runaway spending keeps accruing while you investigate. Stop the spend first, then find out what it was doing.

A health endpoint that returns 200 because the process is running tells you the process is running. Interviewers want to hear where you draw the line between a check that pages and a check that removes traffic.

Code rolled back and the database did not. The old version is now running against a schema it has never seen, and that is a worse incident than the one you were trying to end.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

An idle database with a starved application is the tell. The connections are held, not used. More database capacity does not release leaked connections.

Status codes describe an HTTP failure category. Locate the component that emitted the response, then use its logs to distinguish connection, capacity and application failures.

Three places, and which one depends on how the thing was started. Knowing which logs the runtime collects and which the application writes itself, and that journald may not persist across reboots, saves the first twenty minutes.

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

The gentlest scheduling question there is, and it still separates people who read the events from people who guess. Check whether the pod has a node before choosing between scheduling and startup diagnosis.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.

Tests whether you know that Kubernetes waits, and for how long. The default timings surprise most candidates, and they explain why a node failure hurts longer than people expect.

Intermittent DNS is the most-reported and least-understood Kubernetes failure. There are three well-known causes and each leaves a different fingerprint.

Tests the mitigate-before-diagnose reflex under time pressure, plus whether you know the two ways a disk can be full and the trap that makes freed space not come back.

Both can be right, which is the whole point. Retransmissions mean a segment was not acknowledged, and the network is only one of the places that can happen.

The classic gap between what the backend measures and what the user experiences. A 504 is emitted by something in front of the backend, which is exactly the part your service dashboards do not cover.

Automation can renew a certificate without deploying it correctly. Check the served leaf and chain, the client trust store and the exact verification failure.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

Established traffic keeps working while new connections die at random, which is exactly why this one confuses people. The table is full of flows that will not be needed again for days.