Amazon & AWS Incident Response & Production Debugging interview questions

Tests the mitigate-before-diagnose reflex under time pressure, plus whether you know the two ways a disk can be full and the trap that makes freed space not come back.

Automation can renew a certificate without deploying it correctly. Check the served leaf and chain, the client trust store and the exact verification failure.

Established traffic keeps working while new connections die at random, which is exactly why this one confuses people. The table is full of flows that will not be needed again for days.

Same-node good, cross-node bad is not a random clue, it is nearly a diagnosis. One short decision tree separates overlay reachability, encapsulation blocks, MTU and policy.

Bandwidth graphs look healthy and the partner swears they are fine. The wall is ports, not bits: the available source-port budget depends on the gateway and its address allocation.

Mixed resolution after a transfer is not propagation voodoo. Compare the parent delegation, authoritative answers and validation results. NXDOMAIN and SERVFAIL are clues; neither names a unique cause.

Stateful firewalls keep established flows alive after their rule vanishes, so cleanup day passes every check and lag appears days later. Flow logs name the hour; inventories prevent the repeat.

Two commands disagreeing is the diagnosis, not the mystery. A deleted file with an open handle is invisible to du and fully charged by df, and rm made it worse rather than better.

The volume is in one availability zone and the scheduler cannot move it. Spare capacity in the other two zones is not spare capacity for this pod, and no amount of scaling helps.

Two controllers with different opinions and no shared hysteresis. Flat traffic with an oscillating node count means the loop is driven by something other than demand, and the candidates are few.

Code rolled back and the database did not. The old version is now running against a schema it has never seen, and that is a worse incident than the one you were trying to end.

Every SDK fetches its credentials from a link-local metadata endpoint with a modest request budget. Enough containers refreshing at once and the endpoint starts refusing, which the SDK reports as an auth problem.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

An idle database with a starved application is the tell. The connections are held, not used. More database capacity does not release leaked connections.

Status codes describe an HTTP failure category. Locate the component that emitted the response, then use its logs to distinguish connection, capacity and application failures.

Three places, and which one depends on how the thing was started. Knowing which logs the runtime collects and which the application writes itself, and that journald may not persist across reboots, saves the first twenty minutes.

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

The gentlest scheduling question there is, and it still separates people who read the events from people who guess. Check whether the pod has a node before choosing between scheduling and startup diagnosis.