Databricks SRE / Platform Engineer interview questions

The swap-used graph is the least informative memory metric on the box. Here is what swappiness actually weights, which numbers to watch instead, and the policy worth defending.

Multiplexing removes application-level queuing and hands the problem to TCP. Here is where the blocking really lives, the loss math behind it, and the four levers that help.

Absolute caps versus latency-target protection, systemd IO attributes versus raw io.max, and why ionice lies to you. The staff-level answer assigns the job its own slice with intent.

Two structural differences matter more than any service mapping: the VPC is global rather than regional, and permissions are granted to members on resources through a hierarchy rather than attached to identities.

Large object uploads are a reliability design problem, not a bandwidth one. Parts, retries and the completion call decide whether a flaky link costs one chunk or the whole night.

Queue, log and router solve different problems that all get called events. Picking by workload shape beats picking by whichever service the team already knows.

Quotas are invisible until launch day. Treating them as tracked inventory with utilization alerts beats discovering them in a postmortem.

A cluster boundary does not always create a new cloud identity. Inspect the workload pool and principal identifier before assuming a leaked key.

General-purpose IaC earns its complexity where application engineers own infrastructure and benefit from typed components and unit tests. It disappoints when the pitch was freedom: the engine still builds a graph, still keeps state, and still forbids most of the clever runtime logic people expect to write.

Provider upgrades are a scheduled platform product: automatic version inventory, a canary stack inside days, automation that fans out pinned bumps with plan output attached. Estates without this routine get upgraded by deprecation deadlines.

Registries live or die on upgrade ergonomics, not on publishing features. Green example plans gate every tag, automation delivers upgrade PRs with plan output to consumers, and retirement is a census plus a date.

The pitch is continuous reconciliation instead of a plan someone runs. The cost is that your cloud estate now depends on a cluster, and that drift correction without a plan step is a very different risk posture.

A mock can confirm that a module requests private storage while missing the IAM denial that stops creation. Separate configuration contracts from provider behavior, and make the costly tests deliberate.

The single most common Kubernetes interview question, and the one where a vague answer is most obvious. Every stage you name is a place the interviewer can ask you to debug.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

Almost always a finalizer, and the fast fix is the one that loses data. The safe answer explains what the finalizer is protecting against before removing it.

The scheduler is behaving correctly, which is the part that confuses people. It optimises placement at admission time and never revisits, so a cluster drifts out of balance by design.

A specific configuration trap with a clean explanation. Both controllers act on the same signal from opposite directions, which can produce unstable feedback depending on their policies and workload.

Default-deny egress breaks name resolution before it breaks anything else, and every connection times out confusingly. The practitioners are the ones who test with positive AND negative probes and know their CNI's fine print.

The two ideas that decide whether an operator is safe are level-triggered reconciliation and finalizer semantics. The partition question separates people who have written a controller from people who have installed one.

Namespaces partition resources, not trust. This question finds the people who know which sysctls are namespaced, which ones change every tenant at once, and when only a second kernel will do.

Multiple schedulers are supported, rarely understood, and dangerous in a specific way. The strong answer names the deadlock that gang scheduling fixes and the capacity race it can open.

Repeated full-resource polling can make an operator an expensive API client. This question separates people who run controllers at ten-thousand-object scale from those who copied a tutorial: watch semantics, cache behaviour, and which LISTs touch etcd.

SSH plus yum is how outages start. The senior answer replaces nodes from versioned images on a cadence, orchestrates with surge capacity and PDBs, and can explain the kernel-skew problem that makes fleet homogeneity a feature.

Two artifacts hide under 'Kubernetes backup' and they recover different things. The misses are what decides whether a restore works: volume bytes, encryption keys, and everything that never lived in etcd at all.

Tenancy is a dial from policy-based soft isolation to hard cluster boundaries with a virtual-control-plane middle ground. The scored answer picks by blast radius requirements and names what each model still leaves shared.

Driver, container toolkit, device plugin, DCGM and discovery must version-match on every node. The operator reconciles that ladder from one ClusterPolicy CR; MIG, upgrades and host kernels are where it bites.

Changing pod resources historically required recreation, which made vertical autoscaling disruptive for workloads with expensive recovery. In-place resize changes that, with limits worth knowing.

A billion-row backfill is a deployable, throttled, resumable job with checkpoints and verification gates. Ad-hoc scripts against prod are how it becomes an incident.

Build graphs identify affected targets only when their dependencies are complete. The ladder from build-everything to content-addressed caching, hermeticity audits, and where undeclared inputs rot correctness.

For scheduled and stateful workloads the exposure unit is records and time, not requests: shadow runs on real input, row-level diff budgets, checkpoint compatibility, one-cycle warm standby.

Separate blob integrity from action-result trust, isolate cache writers and investigate differences found by sampled cold rebuilds.

Runner migration questions hide two different exams: whether a PR job can steal your cloud credentials, and why the fleet idles at half price all night. Isolation first, economics second.

The code did not change, the data did. That inverts what triggers the pipeline, what the tests assert, and what a rollback means.

Request-shaped SLIs quietly fail on data platforms. The strong answer defines freshness, completeness and deadline SLIs on consumer-visible artifacts, not job success rates.

One disk metric explaining a whole cluster's weirdness is the point of this question. Restarting etcd first turns a slow cluster into a down one, because a restarted member has to catch up before it can vote.

More partitions should mean more throughput, so why did the backlog deepen? Because rebalance behavior depends on the group protocol, and the way consumers divide work decides who pays for it.

Fork failures can reflect PID limits or memory allocation failure even when host RAM looks available. Find which ceiling broke, cap PIDs per cgroup so one team's leak costs one pod, and fix the unbounded spawn.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

A reported successful rebuild still leaves several places to investigate: a cache, a routing pointer, a chunk that survived deletion, or a document that never reached the pipeline at all.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

The disk is not full, the keyspace quota is. Defragmenting without disarming the alarm leaves the cluster read-only, and disarming without compacting puts it back within minutes.

Two controllers with different opinions and no shared hysteresis. Flat traffic with an oscillating node count means the loop is driven by something other than demand, and the candidates are few.

Every SDK fetches its credentials from a link-local metadata endpoint with a modest request budget. Enough containers refreshing at once and the endpoint starts refusing, which the SDK reports as an auth problem.

Escapes ride kernel bugs, privileged configuration and host mounts, not namespace magic. Cut the syscall and capability surface first, sandbox only what the trust boundary demands, and build nodes expecting compromise.

Trace a retry through the broker, database and external provider. The guarantee changes at each boundary, and a recorded intent does not prove a payment or email completed.

Cron fires at wall-clock times; a scheduler reasons about data intervals and blocked frontiers. The scored details are per-partition idempotency and catch-up semantics.

Teams need comparable evaluations before they approve a model change. Design versioned test data, recorded runs and release thresholds that a pipeline can enforce.

A single FIFO queue is a first-come-first-served promise to whoever submits fastest. Fairness needs per-tenant queues and a scheduler that picks between them, and priority without fairness just moves the problem.

A shared cluster produces one bill and no answer. Attribution is a measurement problem first and a cultural one second, and the number that changes behaviour is not the one finance asked for.

The difference between scaling node groups and provisioning nodes is the whole answer. One picks from shapes you defined in advance; the other reads the pending pods and buys what they need.

Everyone quotes the discount. Fewer candidates can describe what happens when a notice arrives late or a worker disappears without completing shutdown and why some workloads shrug it off while others lose hours of compute.

Refuse every exception and teams leave the platform without telling you. Grant them freely and you are patching nine stacks. The detail that decides it is whether the exception has an expiry date.

Complaints measure volume and confidence, not cost. The loudest team is usually the one most comfortable complaining, which is not the same as the one losing the most time.

You cannot run the counterfactual. What survives scrutiny is a comparison design that narrows the alternatives, not a number that claims to settle it.

Every capability a platform accepts is a permanent operational commitment. Teams that cannot say no end up operating six things badly instead of two things well.

Code is one of three inputs now. Data and the trained model are the other two, and they version differently, break differently and cannot be tested the way code is.

A loop that calls a model, gets asked to run a tool, runs it, and feeds the result back. Every operational difference comes from the fact that the number of iterations is not known in advance.

Tokens are the unit and input and output are priced differently. Get the arithmetic on the table in the first minute, because the number usually decides the design before any other consideration does.

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.

Code rollouts ask 'does it work'; model rollouts ask 'is it better', and the second question has no HTTP status code. Shadow traffic, offline eval gates and quality canaries replace the 500-rate dashboards you cannot use.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Both workloads are legitimate and they want opposite things from the same GPU. Combine tenant rate limits with fair admission and measured capacity reservations; separate fleets are an option when shared scheduling cannot meet the targets.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Asking nicely in the prompt gets you most of the way and fails on the tail, which is where the automation breaks. Constrained decoding can guarantee the shape of completed output for supported schemas. Interrupted generations still need a failure path, and compilation has a latency cost.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

Scale to zero is correct for some workloads and an outage generator for others, and the deciding number is what the first request after idle experiences. Cold start turns an availability promise into a queue.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

A training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.

The training part is a job. Everything around it is the pipeline: data snapshots that are immutable, an evaluation gate with a threshold set in advance, and adapters versioned so a bad one is a routing change away from gone.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Code and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.

A model is an artifact with a lineage, an evaluation record and an owner, or it is a file somebody copied to a bucket. The registry exists so that the second one cannot reach production.

Before blaming the model, check whether it is being fed the same thing it was trained on. Most of the gap is a pipeline difference, and pipeline differences are a platform problem with a platform fix.

The database part is easy. The operational problems are reindexing when the embedding model changes, recall degrading silently as filters get stricter, and memory growing faster than anyone forecast.

Per-token pricing looks expensive until you compute what an idle GPU costs at 3am. Break-even depends on workload volume and duty cycle, measured throughput, and the engineering cost on both sides.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

Nobody is waiting, which changes every constraint. Latency stops mattering, throughput per dollar becomes the only metric, and the design converges on something that looks like a data pipeline rather than a service.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

Shared batching puts two tenants' data in one process on one device. That is defensible, and it is not what most customers assume they bought, so the answer has to be specific about which boundary exists where.

Explain an AI feature failure through its promised outcome, observed quality gap, bounded recovery and the evidence needed for another release.

Decide whether to continue a migration using remaining costs and current benefits, then deliberately operate or unwind the partial state.

Multi-AZ is the answer and the reason matters more than the acronym. Each tier fails differently, and the database is the one where high availability means a failover with a real gap, not zero interruption.

One is layer 4 and regional, one is layer 7 and regional, one is layer 7 and global. Answer on those two axes and the choice makes itself, including when you need two of them together.

A pod is a shared network and storage boundary that happens to contain containers. Once you can say what the containers inside one share, sidecars and init containers stop needing separate explanations.

Service types choose how an application is exposed. Follow the actual traffic path: cloud controllers can target nodes or pods, and internal exposure differs from public exposure.

A Service moves connections and an Ingress reads requests. The gap between those two sentences is hostname routing, path routing and TLS, and it is also why an Ingress object alone does nothing.

The Indian e-commerce interview question, and autoscaling is the wrong first answer. A 10x spike arriving in seconds outruns any scaling loop, so the design has to absorb rather than scale.

Four levels, and the two that matter operationally are the subscription, where quotas and billing live, and the resource group, which is the unit you delete. Getting the boundaries wrong is expensive to unpick later.

Service principal with a client secret is the answer people give and the one that leaves a credential in a config file. Managed identity is the same object with the secret handled by the platform.