Microsoft SRE / Production Engineer interview questions

The answer is yes, and then the real question starts. Asymmetric routing and silent drops punish anyone who treats the routing tables as one flat list.

The signature is unmistakable once seen: handshakes complete, then the connection freezes at the same byte count every time. Here is the probe sequence and the fixes, ranked.

Table full is the symptom everyone knows. The interview answer lives one layer down: per-state timers, the five-day ESTABLISHED default, and what conntrack -S says about hash pressure.

Four levels, and the two that matter operationally are the subscription, where quotas and billing live, and the resource group, which is the unit you delete. Getting the boundaries wrong is expensive to unpick later.

Service principal with a client secret is the answer people give and the one that leaves a credential in a config file. Managed identity is the same object with the secret handled by the platform.

One is layer 4 and regional, one is layer 7 and regional, one is layer 7 and global. Answer on those two axes and the choice makes itself, including when you need two of them together.

The comparison table is easy to recite and explains nothing about the failure the difference actually causes. The ephemeral port range is the whole answer.

Most candidates can sell serverless. The question asks you to argue against it, which needs a real understanding of the execution model rather than the marketing.

A question that sounds introductory and turns on one distinction: what you stop operating, and what you start paying for instead. The answer is three mechanical differences and one economic one.

Four options with genuinely different costs, and the requirement is usually narrower than it first sounds. Establish whether it is about the network path, the data residency, or the provider's retention terms, because those need different answers.

The private endpoint can be healthy while an office resolver still returns the public address. Trace the lookup from the failing client before changing access rules.

Moving a Kubernetes service account changes its token subject. Separate pod mutation, federation trust and Key Vault authorization before adding permissions.

General-purpose IaC earns its complexity where application engineers own infrastructure and benefit from typed components and unit tests. It disappoints when the pitch was freedom: the engine still builds a graph, still keeps state, and still forbids most of the clever runtime logic people expect to write.

Distinguish a gated Terraform run workflow from a cloud controller that reconciles continuously. Both can start from Git, but they expose different approval, drift and credential boundaries.

The pitch is continuous reconciliation instead of a plan someone runs. The cost is that your cloud estate now depends on a cluster, and that drift correction without a plan step is a very different risk posture.

Service types choose how an application is exposed. Follow the actual traffic path: cloud controllers can target nodes or pods, and internal exposure differs from public exposure.

A Service moves connections and an Ingress reads requests. The gap between those two sentences is hostname routing, path routing and TLS, and it is also why an Ingress object alone does nothing.

Selectors match object labels; owner references record controller relationships. Keeping those mechanisms separate explains how a Service can select the wrong pods without owning them.

It sounds like trivia and screens for something real: whether you understand that a pod is a shared namespace boundary with an owner, not just a group of containers.

SSH plus yum is how outages start. The senior answer replaces nodes from versioned images on a cadence, orchestrates with surge capacity and PDBs, and can explain the kernel-skew problem that makes fleet homogeneity a feature.

Kubelet resource metrics to metrics-server to aggregated API to HPA controller: every hop adds lag and averaging. Walking the path and naming where it distorts demand separates operators from dashboard readers.

Heartbeats, conditions and taints are the only vocabulary the control plane understands. NPD translates journald and kernel ring messages into that vocabulary; custom plugins carry your hardware's dialect.

Kubernetes does not see GPUs, it sees an extended resource that something has to advertise. When that advertisement is missing or wrong, the scheduler is behaving correctly and the node is lying.

A Job with a sidecar was broken by design for years: the workload finishes, the proxy does not, and the pod never terminates. Native sidecars fix it by making the container an init container that keeps running.

Changing pod resources historically required recreation, which made vertical autoscaling disruptive for workloads with expensive recovery. In-place resize changes that, with limits worth knowing.

Ingress solved routing by convention and annotation, which meant every controller invented its own. Gateway API's real contribution is not features, it is splitting one object into three so that platform and application teams stop editing the same file.

Clusters multiply for good reasons: regions, isolation, tenancy, blast radius. The problem is not creating them, it is that cluster seven diverges from cluster three and nobody knows until an upgrade behaves differently.

Route attachment and cross-namespace backend access have different owners. Use route status to find the failing boundary before broadening any permission.

Four nested things and one rule that explains most confusion: jobs run on different machines and share nothing unless you make them. Everything about artifacts and caching follows from that.

Stages, jobs, steps, and a pool that supplies the machine. The piece that decides most real designs is the agent pool, because Microsoft-hosted agents cannot reach a private network and most enterprise deployments need to.

Git merges by comparing both branches against their common ancestor. A conflict means both sides changed the same region, and the resolution nobody checks is the one that compiles and is semantically wrong.

Follow commit identities through a rebase, recover a collaborator’s work with an explicit boundary, and explain the limits of force-with-lease and revert.

Follow scheduling eligibility, job isolation and artifact transfer, then diagnose a queued job whose available runners cannot satisfy its requirements.

Deployment and release are two decisions, not one. Choose stable cohorts, test failure defaults and propagation, and retire temporary flags without breaking older application versions.

Quarantine is a holding cell with an expiry date, not a landfill. Per-test flake rates, entry evidence, an enforced expiry review, and the cap that turns fixing into priority.

Green PRs can still break main because they were validated against a stale tip. Merge queues revalidate the exact combination that lands, trading extra validation work for coverage of combined changes.

Define the five delivery measures from explicit events, distinguish failure from rework, and expose uncertain attribution and small samples.

Build graphs identify affected targets only when their dependencies are complete. The ladder from build-everything to content-addressed caching, hermeticity audits, and where undeclared inputs rot correctness.

One cadence for every team is the naive answer. Cadence follows constraints: store review for mobile, compatibility windows for APIs, contracts and embedded realities for trains.

The code did not change, the data did. That inverts what triggers the pipeline, what the tests assert, and what a rollback means.

A token naming the right repository does not prove that the approved deployment workflow ran. Bind the caller, deployment context and called workflow into the cloud trust contract.

MTTR is three clocks hiding in one number. Profile detection, diagnosis and fix separately from incident timestamps; attack whichever clock dominates, which is almost never rollback speed.

An SLA is a price list for failure signed before the failures. Bring the measured distribution, negotiate methodology and exclusions harder than the headline number, and phase targets with dates.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.

Emergency mode after a storage change is usually one line in fstab. The order of operations is what decides the outcome here, along with whether the recovery preserves data-mount dependencies rather than booting past them.

Load balancing makes a per-host fault look like a flaky library. The exact rejection string tells you which way the offending clock leans before you log into anything.

Gradual drift produces no errors on the host that has it, only weirdness everywhere else. Sweep first, histogram second, repair third: the distribution shape names the culprit before you log into anything.

Bandwidth graphs look healthy and the partner swears they are fine. The wall is ports, not bits: the available source-port budget depends on the gateway and its address allocation.

Purges act on exact cache keys, and the edge is full of variants the purge never named. Verify per variant at real POPs, then fix the key model instead of chasing URLs forever.

Stateful firewalls keep established flows alive after their rule vanishes, so cleanup day passes every check and lag appears days later. Flow logs name the hour; inventories prevent the repeat.

Wall-clock schedules meet daylight-saving transitions: spring forward deletes an hour of cron targets, fall back repeats one. Schedule in UTC, add idempotency keys, rehearse in staging.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

A reported successful rebuild still leaves several places to investigate: a cache, a routing pointer, a chunk that survived deletion, or a document that never reached the pipeline at all.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

Running pods already have their images, so a credential expiry is invisible until something needs to start. That gap is why this always surfaces during an unrelated incident, at the worst moment.

Threat modelling dies as a document ritual and lives as a design-stage habit. The answer that scores: tier reviews by blast radius, short sessions on RFCs, findings filed where engineers already work.

Push approval is a weak factor and fatigue is its predictable failure mode. Stop the campaign with number matching and rate limits; end the attack class with passkeys, device gating and a hardened helpdesk.

Labels without consequences die quietly. Attach classification to enforced defaults in Terraform modules, propagate tags to every copy of the data, and give engineers twenty worked examples instead of a taxonomy lecture.

An agent is a workload that acts on behalf of a person, holds credentials for several systems, and decides what to call based on text it read. Every one of those properties breaks an assumption in your existing access model.

Model output is attacker-influenced data by construction, because the model reads attacker-influenced input. Every rule you already apply to user input applies here, and teams forget because the text came from a system they built.

The most rehearsed design question, which is exactly why the interesting parts are the ones blog posts skip: redirect semantics, key generation under failure, and analytics kept off the hot path.

The difficult decisions are preserving offline intent and enforcing rules that span devices. Name the conflict classes and handle each deliberately rather than proposing a CRDT framework on instinct, because most of these conflicts have a domain answer that no merge algorithm can supply.

Reciting both acronyms is the median answer. Expert level chooses by connectivity assumptions and undo semantics, then defends CRDT metadata growth with numbers.

Teams need comparable evaluations before they approve a model change. Design versioned test data, recorded runs and release thresholds that a pipeline can enforce.

Queue time and build time are separate problems with separate fixes, and most teams spend a quarter optimising the one that is not hurting. Size the fleet from arrival rate, then attack the build with caching and graph awareness.

Data transfer appears on nobody's architecture diagram and reliably appears on the invoice. The strong answer enumerates the taxed flows, ranks the levers, and knows which tax is worth paying.

An upfront purchase, a refund and two currencies can produce several valid totals. Work out which total finance needs, then prove every adjustment from the source rows.

Refuse every exception and teams leave the platform without telling you. Grant them freely and you are patching nine stacks. The detail that decides it is whether the exception has an expiry date.

A generated template is a copy, not a link. Drift is not a discipline failure, it is the predictable result of the mechanism you chose.

Complaints measure volume and confidence, not cost. The loudest team is usually the one most comfortable complaining, which is not the same as the one losing the most time.

You cannot run the counterfactual. What survives scrutiny is a comparison design that narrows the alternatives, not a number that claims to settle it.

Saying yes without a contribution contract produces features the platform team did not design and now supports forever, on call, at 3am.

Code is one of three inputs now. Data and the trained model are the other two, and they version differently, break differently and cannot be tested the way code is.

A loop that calls a model, gets asked to run a tool, runs it, and feeds the result back. Every operational difference comes from the fact that the number of iterations is not known in advance.

Tokens are the unit and input and output are priced differently. Get the arithmetic on the table in the first minute, because the number usually decides the design before any other consideration does.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

The driver, the container toolkit, the CUDA runtime inside the image and the framework build all have to agree. Upgrade the wrong one first and every pod on the node fails to start with an error that names none of them.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

At sixteen GPUs and above, the network is the computer. Collectives run at the speed of the worst link in the group, and placement decides which link that is.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

Scale to zero is correct for some workloads and an outage generator for others, and the deciding number is what the first request after idle experiences. Cold start turns an availability promise into a queue.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

Prompts contain whatever the user typed, which in a support product is names, account numbers and medical detail. Full-text logging is a data-protection decision, not a debugging preference.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

The training part is a job. Everything around it is the pipeline: data snapshots that are immutable, an evaluation gate with a threshold set in advance, and adapters versioned so a bad one is a routing change away from gone.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

A model is an artifact with a lineage, an evaluation record and an owner, or it is a file somebody copied to a bucket. The registry exists so that the second one cannot reach production.

The database part is easy. The operational problems are reindexing when the embedding model changes, recall degrading silently as filters get stricter, and memory growing faster than anyone forecast.

A release-automation design question with a specific trust boundary. The novel risk is prompt injection: untrusted text reaching a privileged agent, where the text is the instruction and your usual input validation does not apply.

Untrusted text reaching a privileged agent is the whole vulnerability. It is the pull_request_target mistake again, with a model in the middle that will follow instructions it finds in the data.

The valuable part is the first ten minutes of context gathering, which is tedious and mechanical. The dangerous part is anything that changes production while a human is still working out what is wrong.

Generated infrastructure code is plausible, fluent and confidently wrong in specific ways. The volume goes up, the reviewer attention per line goes down, and the controls that used to be optional stop being optional.

Every team standing up its own tool server produces a permission surface nobody can enumerate. The gateway exists so that connecting an agent to a system is a reviewed act rather than a config file in somebody's repository.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

GPUs are the default for reasons that hold for large generative models and not for everything else. A small classifier on CPU can be cheaper, more available and easier to operate, and knowing where the crossover sits is worth more than knowing the newest card.

A model artifact is code that runs in your cluster with GPU access. Most of the supply-chain controls you already run for containers apply, and the file format adds one of its own.

Shared batching puts two tenants' data in one process on one device. That is defensible, and it is not what most customers assume they bought, so the answer has to be specific about which boundary exists where.

A mandatory gateway can centralize limits and filtering, while applications authorize business actions. Neither a prompt nor a PII classifier guarantees that sensitive data stays private.

Most of the obligation lands on the product, but three parts land on infrastructure: an inventory of what is deployed, evidence of what it did, and the ability to turn it off. Those are platform capabilities and they take months to retrofit.

Facilitate a disputed design by stopping personal attacks, separating testable assumptions from priorities and assigning a decision owner.

Delegate a technical outcome with authority, capacity and acceptance criteria, while preserving the decisions and commitments assigned to the lead.

Turn difficult feedback into a testable improvement, while checking its factual basis and keeping unfair or inappropriate treatment separate.

Explain an AI feature failure through its promised outcome, observed quality gap, bounded recovery and the evidence needed for another release.

Retire an internal tool through verified user needs, compatibility evidence and an owned cutover, while treating its maintainer as a source of context rather than a barrier.

Handle a slipping dependency through a tested interface, honest integration status and an early decision about scope, resources or the external commitment.

Multi-AZ is the answer and the reason matters more than the acronym. Each tier fails differently, and the database is the one where high availability means a failover with a real gap, not zero interruption.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

Experienced answers are ordered and verifiable: minimal base, benchmark applied as code, identity and exposure next, then continuous proof it stayed that way. Checklists recited out of order are the tell.

A pod is a shared network and storage boundary that happens to contain containers. Once you can say what the containers inside one share, sidecars and init containers stop needing separate explanations.

Compare proxy, hosted and group repositories. Trace a tested artifact into production, and understand when a cache can keep builds running during an upstream outage.

Follow a Deployment change through the API server, controllers, scheduler and kubelet. Each handoff gives you a different place to investigate when a pod never starts.

The single most common Kubernetes interview question, and the one where a vague answer is most obvious. Every stage you name is a place the interviewer can ask you to debug.