Datadog Incident Response & Production Debugging interview questions

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Intermittent DNS is the most-reported and least-understood Kubernetes failure. There are three well-known causes and each leaves a different fingerprint.

The classic gap between what the backend measures and what the user experiences. A 504 is emitted by something in front of the backend, which is exactly the part your service dashboards do not cover.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

Nothing errors because dropping unmatched lines is documented behaviour. Counting the tagged failures converts an invisible leak into a number, and numbers have owners.

The kill was not random and it was not fair. Learn the scoring function the kernel actually uses, then bend it with oom_score_adj before your monitoring dies during the next leak.

Spikes that march in step with BGSAVE are a page-size story, not a network story. Check actual huge-page usage, fork duration and copy-on-write pressure on the deployed kernel.

Slow leaks hide inside restart cycles and percentage thresholds. Plot weeks of absolute memory against deploys, prove it with profiler deltas, and alert on predicted time-to-limit instead.

Raising the limit buys hours and proves nothing. The six-hour periodicity is the diagnosis: something is opening descriptors on a code path that never closes them.

Two controllers with different opinions and no shared hysteresis. Flat traffic with an oscillating node count means the loop is driven by something other than demand, and the candidates are few.

Somebody silenced it during an incident nine days ago to stop the noise, and nothing ever expired the silence. The monitoring was working perfectly and nobody was listening.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

An idle database with a starved application is the tell. The connections are held, not used. More database capacity does not release leaked connections.

Status codes describe an HTTP failure category. Locate the component that emitted the response, then use its logs to distinguish connection, capacity and application failures.

Three places, and which one depends on how the thing was started. Knowing which logs the runtime collects and which the application writes itself, and that journald may not persist across reboots, saves the first twenty minutes.

The gentlest scheduling question there is, and it still separates people who read the events from people who guess. Check whether the pod has a node before choosing between scheduling and startup diagnosis.

Tests whether you know that Kubernetes waits, and for how long. The default timings surprise most candidates, and they explain why a node failure hurts longer than people expect.