SRE, Prometheus & Observability Interview Questions

Prometheus asks your service for numbers on a schedule rather than waiting to be told. That one design choice explains service discovery, exporters, staleness and why short-lived jobs are the awkward case.

Metrics quantify impact, traces expose instrumented request paths, and logs describe individual events. Follow a slow checkout across all three while accounting for missing evidence.

All three can search logs. What differs is who operates the cluster and how the bill is shaped, and the bill is what kills these projects, not the querying.

The arithmetic takes ten seconds and is the part most often skipped. The stall is the signal: it means error budgets are something you have read about rather than used.

Defining chaos engineering is easy. Far fewer can say what they would break first, and almost none volunteer the abort condition, which is the thing that makes it a programme rather than an incident.

The observability failure that scales with your success. Cardinality is the cost driver in every metrics system, and the fix has to be structural or it comes back next quarter.

Alert fatigue is a design failure, not a discipline failure. The fix is a rule about what may page at all, and burn-rate alerting is how you implement it.

A question with a right answer and a subtle reason. Histogram versus summary is the part that catches people, and it turns on whether you can aggregate across instances.

Tracing is the only telemetry that answers where the time went across services. The mechanism is context propagation, and the reason teams fail at it is that one uninstrumented hop breaks the chain.

The instinct is to delete old logs. That is backwards: the waste lives in what you ingest, not how long you keep it. This answer walks the levers in the order that actually pays.

OpenTelemetry is an API and a wire protocol, not a product. Candidates who promise instant multi-vendor bliss have never run the collector in production or negotiated with a vendor who knows it.

Day-one telemetry is cheap and retrofitting it costs an incident. The answer that scores names the linking layer (trace IDs in logs, exemplars on metrics) that turns three tools into one story.

The pull model is the famous part and explains none of the operational pain. The storage engine is where Prometheus hurts in production: crash recovery, cold queries, disk death.

Anyone can divide bad requests by total. This answer shows the aggregation mistakes that quietly corrupt SLO dashboards, the rate-versus-increase traps that make budgets lie, and when subqueries cost too much.

Dashboard sprawl is an operating-model problem wearing a tooling costume. The scored answer moves definitions into git, attaches ownership metadata to every panel, and deletes without ceremony.

The data-path question separates people who have sized a cluster from people who installed one. Shard sizing, refresh versus flush, and each ILM phase carry one specific failure mode apiece.

Choose what evidence must survive, then price the collection path. A tail policy can only inspect traces that reach it; low SDK sampling changes the guarantee before the collector sees a span.

Metrics describe the server's experience; synthetics describe the user's. The strong answer designs a few money-path journeys from real geographies, pages only on persistent failure, and budgets for script rot.

The roles question is really a delegation test: can you split commanding the incident from fixing it? Name the roles, the comms cadence, and the handoff discipline that keeps 3am coherent.

Spending the budget is arithmetic; enforcing consequences is politics plus automation. The answer that lands names trigger thresholds, the deploy gate that blocks releases, and the dated escape valve.

The kernel watches syscalls, sockets and short-lived processes that userspace agents polling every few seconds never meet. Know what eBPF adds, what it costs, and the security conversation it starts.

Forty healthy islands equal one blind estate. Compare the consolidation patterns honestly, keep alerting local, curate what crosses the WAN, and make the global view read-only and cheap.

Request-shaped SLIs quietly fail on data platforms. The strong answer defines freshness, completeness and deadline SLIs on consumer-visible artifacts, not job success rates.

Routing rots because orgs change faster than configs. The senior answer treats ownership as data resolved from a catalog at fire time, with drift detection on the routing itself.

Usable runbooks live one click from the alert, front-load symptom-to-action pairs, and stay alive because CI and drills force the issue rather than anyone hoping.

Rotation length decides focus versus fatigue, handoff quality decides whether Tuesday inherits Monday's mess, and follow-the-sun only works when every region holds real authority.

Facilitation is mechanics: timeline before theories, blame converted into system questions, actions with owners tracked to closure. Include the defensive engineer and the exec who wants a head.

Where linear models break: compounding growth, step functions, peak days and saturation cliffs. Forecast per constrained resource, backtest, subtract lead time to get the real order-by date.

Attribution is an incentive-design problem: own-team labels at ingest, unit pricing per signal, monthly showback before any chargeback, sanctioned levers so teams can actually respond.

Detectors earn their keep on seasonal aggregates and fleets of similar series, run in shadow mode with measured precision. They lie on low-volume series, bursty workloads and regime changes.

One surface cannot serve the responder and the review. Tier dashboards by the question each audience asks, and make every tile answer in five seconds or link one click deeper.

MTTR is three clocks hiding in one number. Profile detection, diagnosis and fix separately from incident timestamps; attack whichever clock dominates, which is almost never rollback speed.

Internal platforms hide a trap: the direct user is an engineer, but the real traffic belongs to their customers. Define SLIs from the consumer's vantage and turn the error budget into roadmap currency.

Their green page, your red SLO. Separate their true failure rate from the amplification your own client adds, buffer at the boundary, then escalate with a client-side scorecard instead of feelings.

Sampling keeps proportions and loses rare individuals; filtering keeps chosen classes complete and loses whatever you misclassified. Decide retention at write time, keep counters always, rehydrate the rest.

Assert on the shape of a trace: which spans ran, in what order, with which attributes. Catches the cross-service regressions unit and contract tests structurally miss, at the price of instrumentation discipline.

Monitoring dies silently: stuck alert queues, full TSDBs, collectors killed inside the same blast radius. Layer independent checks and a heartbeat so absence of signal pages louder than any red dashboard.

Severity inflation and silent underdeclaration are both incentive problems, not wording problems. Anchor levels to impact and obligations, pre-write example scenarios, and make declaring severe safe.

Comms is a role, not a chore bolted onto the incident commander. Audience-specific cadences, a committed next-update time in every message, and plain language that support can actually repeat.

An SLA is a price list for failure signed before the failures. Bring the measured distribution, negotiate methodology and exclusions harder than the headline number, and phase targets with dates.

You have added a dependency with a latency distribution an order of magnitude wider than anything else in your stack, whose failure mode is a plausible wrong answer rather than an error.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

Tracing tells you which service and which span. It stops at the function boundary, and that is exactly where a CPU regression lives. Continuous profiling is the layer most teams do not have when they need it.

There is no status code and no caller waiting, so latency and error rate stop meaning what they meant. The signals that replace them are age, lag and completion, and they answer a question the request-response set never had to ask.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.