PhonePe Incident Response & Production Debugging interview questions

Status codes describe an HTTP failure category. Locate the component that emitted the response, then use its logs to distinguish connection, capacity and application failures.

Both can be right, which is the whole point. Retransmissions mean a segment was not acknowledged, and the network is only one of the places that can happen.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

One disk metric explaining a whole cluster's weirdness is the point of this question. Restarting etcd first turns a slow cluster into a down one, because a restarted member has to catch up before it can vote.

The reflex is rollback and it is usually right. Shared state, changed credentials and unavailable old artifacts can make an undo incomplete or unsafe. Verify those dependencies before choosing the recovery.

Spikes that march in step with BGSAVE are a page-size story, not a network story. Check actual huge-page usage, fork duration and copy-on-write pressure on the deployed kernel.

Split-brain writes are a fencing failure first and a data problem second. Freeze writes, read the divergence boundary from replication positions, and let the payment ledger arbitrate reality.

Stop the source before tuning anything: kill the runner, revoke its credentials, shed test-shaped traffic at the edge, protect checkout. Forensics and guardrails only after customers stop feeling it.

Raising the limit buys hours and proves nothing. The six-hour periodicity is the diagnosis: something is opening descriptors on a code path that never closes them.

Running pods already have their images, so a credential expiry is invisible until something needs to start. That gap is why this always surfaces during an unrelated incident, at the worst moment.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

An idle database with a starved application is the tell. The connections are held, not used. More database capacity does not release leaked connections.

Three places, and which one depends on how the thing was started. Knowing which logs the runtime collects and which the application writes itself, and that journald may not persist across reboots, saves the first twenty minutes.

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

The gentlest scheduling question there is, and it still separates people who read the events from people who guess. Check whether the pod has a node before choosing between scheduling and startup diagnosis.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.