Datadog SRE / Platform Engineer interview questions

A growing backlog of sockets waiting for the application to close its side usually points at lifecycle handling. Here is the hunt from ss output to the pool timeout that caused it.

The flags are the easy part. The scored part is proving you can capture on a saturated box without adding load, then read the file afterward like evidence.

Everything top, htop and your APM agent display is derived from procfs anyway. Knowing the raw files turns any bare SSH session into a monitoring console.

tcpdump copies packets to userland; eBPF aggregates inside the kernel next to the event. Know which hook runs where, what a verifier log is telling you, and when XDP drops for free.

X-Forwarded-For, PROXY protocol v1 and v2, and the trust boundaries between them. Get the layer wrong and your rate limiting, geo rules and audit trails all key off the balancer's address.

Anycast turns BGP convergence into your failover mechanism. Sessions stick to the site they landed on, monitoring must go per-site, and a route you cannot withdraw cleanly is worse than DNS.

Strip the marketing and Aurora is a rewrite of the storage layer: the log is the database, committed to a quorum across three AZs. Know when that machinery pays and when it just bills.

Quotas are invisible until launch day. Treating them as tracked inventory with utilization alerts beats discovering them in a postmortem.

Four-hour applies with throttling errors share one root cause more often than not. Diagnose which calls burn the quota, split the state along dependency seams, and touch the parallelism flag last, not first.

Everyone can recite the three. Explain what each failure causes, and why a badly configured liveness probe is more dangerous than none at all.

A practical question with a precise answer, and the size is not the real cost. Pull time on every node, registry storage, and attack surface all scale with it.

Compare the configured Service datapaths and their lookup and update mechanisms. The iptables datapath degrades with service count, and eBPF replaces a linear scan with a map lookup.

Explain per-node eligibility and update behaviour, then verify that a Ready collector is actually delivering logs.

The kubelet is a local reconcile loop with its own event stream, probe workers and eviction brain. The detail worth holding onto is that Ready status can reflect either a reported health failure or missing heartbeats, and those have different causes and different fixes.

Dockershim is gone and the CRI contract decides what a runtime must do. The senior-level part is the shim model: a supervisor process that owns exit codes and stdio so the daemon can die without taking workloads with it.

Repeated full-resource polling can make an operator an expensive API client. This question separates people who run controllers at ten-thousand-object scale from those who copied a tutorial: watch semantics, cache behaviour, and which LISTs touch etcd.

Exec into a distroless image and you get nothing, because there is nothing to exec into. Ephemeral containers inject tooling into a running pod without restarting it, and the security caveats are what interviewers really listen for.

Tenancy is a dial from policy-based soft isolation to hard cluster boundaries with a virtual-control-plane middle ground. The scored answer picks by blast radius requirements and names what each model still leaves shared.

Two cleanup mechanisms hide behind one taint: image garbage collection removes unreferenced images by LRU at threshold crossings, while eviction kills pods. Candidates who conflate them fail this question quietly.

Kubelet resource metrics to metrics-server to aggregated API to HPA controller: every hop adds lag and averaging. Walking the path and naming where it distorts demand separates operators from dashboard readers.

Exit codes, the previous-container log, and a little-known file at /dev/termination-log that carries a container's last words into kubectl output. Scale triage starts with classification, not with opening pods.

Best-effort objects, one-hour TTL, deduplicated by count, invisible unless shipped out. Events narrate scheduling and lifecycle decisions; treating them as durable records is the expensive mistake.

Route attachment and cross-namespace backend access have different owners. Use route status to find the failing boundary before broadening any permission.

Define the five delivery measures from explicit events, distinguish failure from rework, and expose uncertain attribution and small samples.

Dependency bots fail as noise generators or silent auto-mergers. Tiering by risk, weekly batching, and treating bot PRs with a real review SLA is what keeps updates flowing.

Prometheus asks your service for numbers on a schedule rather than waiting to be told. That one design choice explains service discovery, exporters, staleness and why short-lived jobs are the awkward case.

Metrics quantify impact, traces expose instrumented request paths, and logs describe individual events. Follow a slow checkout across all three while accounting for missing evidence.

All three can search logs. What differs is who operates the cluster and how the bill is shaped, and the bill is what kills these projects, not the querying.

The arithmetic takes ten seconds and is the part most often skipped. The stall is the signal: it means error budgets are something you have read about rather than used.

Defining chaos engineering is easy. Far fewer can say what they would break first, and almost none volunteer the abort condition, which is the thing that makes it a programme rather than an incident.

The observability failure that scales with your success. Cardinality is the cost driver in every metrics system, and the fix has to be structural or it comes back next quarter.

A question with a right answer and a subtle reason. Histogram versus summary is the part that catches people, and it turns on whether you can aggregate across instances.

Tracing is the only telemetry that answers where the time went across services. The mechanism is context propagation, and the reason teams fail at it is that one uninstrumented hop breaks the chain.

The instinct is to delete old logs. That is backwards: the waste lives in what you ingest, not how long you keep it. This answer walks the levers in the order that actually pays.

OpenTelemetry is an API and a wire protocol, not a product. Candidates who promise instant multi-vendor bliss have never run the collector in production or negotiated with a vendor who knows it.

The pull model is the famous part and explains none of the operational pain. The storage engine is where Prometheus hurts in production: crash recovery, cold queries, disk death.

Choose what evidence must survive, then price the collection path. A tail policy can only inspect traces that reach it; low SDK sampling changes the guarantee before the collector sees a span.

The kernel watches syscalls, sockets and short-lived processes that userspace agents polling every few seconds never meet. Know what eBPF adds, what it costs, and the security conversation it starts.

Routing rots because orgs change faster than configs. The senior answer treats ownership as data resolved from a catalog at fire time, with drift detection on the routing itself.

Attribution is an incentive-design problem: own-team labels at ingest, unit pricing per signal, monthly showback before any chargeback, sanctioned levers so teams can actually respond.

Detectors earn their keep on seasonal aggregates and fleets of similar series, run in shadow mode with measured precision. They lie on low-volume series, bursty workloads and regime changes.

One surface cannot serve the responder and the review. Tier dashboards by the question each audience asks, and make every tile answer in five seconds or link one click deeper.

Sampling keeps proportions and loses rare individuals; filtering keeps chosen classes complete and loses whatever you misclassified. Decide retention at write time, keep counters always, rehydrate the rest.

Monitoring dies silently: stuck alert queues, full TSDBs, collectors killed inside the same blast radius. Layer independent checks and a heartbeat so absence of signal pages louder than any red dashboard.

You have added a dependency with a latency distribution an order of magnitude wider than anything else in your stack, whose failure mode is a plausible wrong answer rather than an error.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

Tracing tells you which service and which span. It stops at the function boundary, and that is exactly where a CPU regression lives. Continuous profiling is the layer most teams do not have when they need it.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Intermittent DNS is the most-reported and least-understood Kubernetes failure. There are three well-known causes and each leaves a different fingerprint.

The classic gap between what the backend measures and what the user experiences. A 504 is emitted by something in front of the backend, which is exactly the part your service dashboards do not cover.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

Nothing errors because dropping unmatched lines is documented behaviour. Counting the tagged failures converts an invisible leak into a number, and numbers have owners.

The kill was not random and it was not fair. Learn the scoring function the kernel actually uses, then bend it with oom_score_adj before your monitoring dies during the next leak.

Spikes that march in step with BGSAVE are a page-size story, not a network story. Check actual huge-page usage, fork duration and copy-on-write pressure on the deployed kernel.

Slow leaks hide inside restart cycles and percentage thresholds. Plot weeks of absolute memory against deploys, prove it with profiler deltas, and alert on predicted time-to-limit instead.

Raising the limit buys hours and proves nothing. The six-hour periodicity is the diagnosis: something is opening descriptors on a code path that never closes them.

Two controllers with different opinions and no shared hysteresis. Flat traffic with an oscillating node count means the loop is driven by something other than demand, and the candidates are few.

Somebody silenced it during an incident nine days ago to stop the noise, and nothing ever expired the silence. The monitoring was working perfectly and nobody was listening.

Signing proves what shipped, not what runs. The strong answer covers syscall-level detection, the signals that matter, tuning against alert fatigue, and the response path once a sensor fires.

Discover, scan, deduplicate, prioritise with context, route with SLA clocks, verify, measure. Most programmes rot at dedup and ownership: five scanners produce five conflicting queues nobody trusts.

Everyone names token bucket. The scored part is the rest: drift between nodes, the 429 contract with clients, and whether your quota service should fail open or closed.

Every overloaded system dies the same way: queues turn load into latency, latency into timeouts, timeouts into retry storms. Tiered shedding, deadline propagation, and retry budgets are the design.

Everyone sizes a metrics system by samples per second and everyone is wrong. The constraint that actually kills these platforms is a single label added on a Friday.

A disputed invoice needs a reproducible calculation. Retain raw usage events and versioned pricing so each line can be explained and corrected.

The hard parts are not search. They are deciding what to keep before you pay to store it, isolating one team's query from everyone else's, and making the cost visible to the people generating it.

The difference between scaling node groups and provisioning nodes is the whole answer. One picks from shapes you defined in advance; the other reads the pending pods and buys what they need.

Everyone quotes the discount. Fewer candidates can describe what happens when a notice arrives late or a worker disappears without completing shutdown and why some workloads shrug it off while others lose hours of compute.

Pay-per-use sounds free until the invoice arrives. The useful method is to decompose the bill into invocations, gigabyte-seconds and the lines nobody models, then locate the honest crossover rather than arguing about it.

An internal platform with no stated objective still has one. Every team guesses it, they all guess differently, and the guesses only surface during an outage.

Operational load is not a discipline problem. It is the interest payment on everything the team shipped without an owner, and it compounds.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

The valuable part is the first ten minutes of context gathering, which is tedious and mechanical. The dangerous part is anything that changes production while a human is still working out what is wrong.

Make early incident reporting practical through a usable escalation path, constructive response and visible corrective work, without turning reports into a performance quota.

Keep incident coverage working while investigating missed notifications, with private conversation, tested escalation and appropriate management support.

Transfer incident responsibility with current impact, evidence-qualified hypotheses, active changes and the next authorized decision, followed by explicit acceptance.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

A pod is a shared network and storage boundary that happens to contain containers. Once you can say what the containers inside one share, sidecars and init containers stop needing separate explanations.

Service types choose how an application is exposed. Follow the actual traffic path: cloud controllers can target nodes or pods, and internal exposure differs from public exposure.

A Service moves connections and an Ingress reads requests. The gap between those two sentences is hostname routing, path routing and TLS, and it is also why an Ingress object alone does nothing.

The Indian e-commerce interview question, and autoscaling is the wrong first answer. A 10x spike arriving in seconds outruns any scaling loop, so the design has to absorb rather than scale.

Follow a Deployment change through the API server, controllers, scheduler and kubelet. Each handoff gives you a different place to investigate when a pod never starts.