Hard DevOps interview questions

The question that reveals whether someone knows what load average actually counts on Linux. It is not a CPU metric, and that single fact resolves the whole thing.

The classic ephemeral port exhaustion question. TIME_WAIT is a familiar phrase and the useful facts are narrower: which side accumulates it, and why the popular fix is dangerous.

Interviewers use this to sort memorised trivia from people who have chased a slow lookup. Three roles, one bit, and the caching rules most candidates repeat incorrectly.

A growing backlog of sockets waiting for the application to close its side usually points at lifecycle handling. Here is the hunt from ss output to the pool timeout that caused it.

The swap-used graph is the least informative memory metric on the box. Here is what swappiness actually weights, which numbers to watch instead, and the policy worth defending.

Latency fine, loss low, link idle, yet a single copy takes hours. The arithmetic of bandwidth-delay product settles this question before any tool opens.

SIGKILL writes no stack trace and the application cannot defend itself. The verdict is already sitting in the kernel ring buffer, and reading it precisely is the skill being priced.

tcpdump copies packets to userland; eBPF aggregates inside the kernel next to the event. Know which hook runs where, what a verifier log is telling you, and when XDP drops for free.

One CPU doing all the packet work is a topology failure, not a tuning mystery. RSS queues, MSI-X vectors, local_cpulist and irqbalance: the whole chain from symptom to pin.

Patching is easy; surviving the rollout is the interview. Staged rings with health gates, what livepatching can and cannot buy you, and the rollback plan you write before starting.

Loss-based versus model-based congestion avoidance decides whether your WAN transfers fill the pipe or die on the first stray drop. Here is the operator's decision, not the RFC summary.

Absolute caps versus latency-target protection, systemd IO attributes versus raw io.max, and why ionice lies to you. The staff-level answer assigns the job its own slice with intent.

Turns the standard autoscaling definition into the problem it actually causes. Almost every candidate can define an HPA; far fewer can say why theirs did not save them.

The question that separates console users from people who have debugged a permissions failure across accounts. Evaluation order is deterministic and knowing it makes AccessDenied readable.

A design question with a clear answer at this scale, and the reasoning is what scores. The three options solve different problems and peering and Transit Gateway connect networks while PrivateLink exposes services.

S3 access failures are rarely about the bucket policy alone. The caller, object ownership and network path determine which authorization checks apply. Start with the failed request and trace changes across those boundaries.

Spreading instances across availability zones is not the same as surviving an AZ failure. What makes the difference is being able to name the four coupling points that quietly re-unify a blast radius you thought you had split.

A busy date key can throttle while table capacity remains available. Check the actual key distribution and throttling reason before adding capacity or shards.

Rapid function scaling can overwhelm a shared dependency. Concurrency limits, reserved capacity and queue buffering are the containment toolkit.

Direct encryption hits a 4 KB wall and a shared throughput quota. Envelope encryption routes around both, and the key policy underneath is central to authorizing key use.

Cross-account access is easy to grant and easy to get wrong. External IDs, scoped roles and audit trails separate a controlled bridge from an open door any tenant can walk through.

Strip the marketing and Aurora is a rewrite of the storage layer: the log is the database, committed to a quorum across three AZs. Know when that machinery pays and when it just bills.

Instance metadata is a local credential vending machine and SSRF is the coin slot. IMDSv2 raises the bar at the host; layered defences and blast-radius limits finish the job.

Residency is not a checkbox on the primary datastore. Where bytes sit, who holds keys, and where logs, backups and analytics quietly replicate decide whether the deal signs.

Blue-green works for stateless apps because instances are disposable. Databases are shared state, so safe migrations ride on backward compatibility and an explicit rollback boundary.

Active-active replication moves writes but not truth. Last-writer-wins quietly discards one of the two orders your finance team expects both to exist.

Data transfer is charged in several unrelated ways and reported as one line, so the first job is decomposition. Most surprises are cross-zone chatter between services that never needed to be on different nodes.

Four options with genuinely different costs, and the requirement is usually narrower than it first sounds. Establish whether it is about the network path, the data residency, or the provider's retention terms, because those need different answers.

A cluster boundary does not always create a new cloud identity. Inspect the workload pool and principal identifier before assuming a leaked key.

The Terraform question that actually filters. Everyone knows both meta-arguments exist; the answer is about how state addresses resources, and why an index is a fragile key.

The services-market question that decides who can actually run a migration. Import is the easy part; the hard parts are resource-for-resource fidelity and proving the first plan is empty.

Drift is a governance problem wearing a tooling costume. Detection is easy; the scored content is attribution (who changed it and why) and the policy decision that most teams avoid making.

sensitive = true stops the echo, not the storage. The complete answer walks the value through git, plan artifacts, state, and provider side effects, naming which mitigation closes which leak.

A green plan is evidence, not proof. Build the pyramid: lint in seconds, policy on every PR, real-cloud verification only where shared modules and money justify the burn.

Stack boundaries follow blast radius and change frequency, not the org chart. The monorepo only pays for itself while a machine can answer, from a commit, exactly which roots need planning.

General-purpose IaC earns its complexity where application engineers own infrastructure and benefit from typed components and unit tests. It disappoints when the pitch was freedom: the engine still builds a graph, still keeps state, and still forbids most of the clever runtime logic people expect to write.

state rm, import, moved and removed blocks each treat a different wound. The commands are the easy part. What makes the operation safe is the backup, lock and plan ritual wrapped around touching production state.

Four-hour applies with throttling errors share one root cause more often than not. Diagnose which calls burn the quota, split the state along dependency seams, and touch the parallelism flag last, not first.

The pitch is continuous reconciliation instead of a plan someone runs. The cost is that your cloud estate now depends on a cluster, and that drift correction without a plan step is a very different risk posture.

A mock can confirm that a module requests private storage while missing the IAM denial that stops creation. Separate configuration contracts from provider behavior, and make the costly tests deliberate.

Evaluate identity, traffic policy and telemetry against measured proxy overhead and operating responsibilities, then test the authorization boundary.

Trace sandbox setup, address allocation and a cross-node packet, then distinguish pod reachability from Service translation.

Almost always a finalizer, and the fast fix is the one that loses data. The safe answer explains what the finalizer is protecting against before removing it.

The scheduler is behaving correctly, which is the part that confuses people. It optimises placement at admission time and never revisits, so a cluster drifts out of balance by design.

Compare the configured Service datapaths and their lookup and update mechanisms. The iptables datapath degrades with service count, and eBPF replaces a linear scan with a map lookup.

Default-deny egress breaks name resolution before it breaks anything else, and every connection times out confusingly. The practitioners are the ones who test with positive AND negative probes and know their CNI's fine print.

Namespaces partition resources, not trust. This question finds the people who know which sysctls are namespaced, which ones change every tenant at once, and when only a second kernel will do.

A registry stores repositories, which hold manifests, which point at content-addressed blobs. Being able to draw that hierarchy and the auth exchange around it accounts for essentially every pull failure, because each one is a specific step in it returning something other than 200.

The loop is easy; the statistics are not. This question finds engineers who can size a traffic slice against sample noise and name the false rollback that kills trust in automation.

Every control plane component depends on the API server, so who starts the API server? The bootstrap paradox sits at the heart of every cluster, and the kubelet plus a directory of files is the accepted way out.

SSH plus yum is how outages start. The senior answer replaces nodes from versioned images on a cadence, orchestrates with surge capacity and PDBs, and can explain the kernel-skew problem that makes fleet homogeneity a feature.

Pod-level SIGTERM handling and node-level shutdown coordination are different mechanisms, and conflating them is the trap. The kubelet can only drain pods on power-off if it participates in systemd's shutdown sequence.

Upgrades look like rolling restarts right up to the morning the CNI stops talking to the new kubelet. The scored answer is ordering, skew policy, and addon sequencing, not courage under pressure.

The classic five-second stall is rarely CoreDNS being slow; it is UDP, conntrack races and ndots expansion. The diagnosis ladder and the NodeLocal DNSCache fix separate operators from readers of architecture blogs.

Three API servers behind one health-checked endpoint, etcd on its own quorum, leader election everywhere else. The scored part is naming what degrades quietly before anything alerts.

Driver, container toolkit, device plugin, DCGM and discovery must version-match on every node. The operator reconciles that ladder from one ClusterPolicy CR; MIG, upgrades and host kernels are where it bites.

Kubernetes does not see GPUs, it sees an extended resource that something has to advertise. When that advertisement is missing or wrong, the scheduler is behaving correctly and the node is lying.

A Job with a sidecar was broken by design for years: the workload finishes, the proxy does not, and the pod never terminates. Native sidecars fix it by making the container an init container that keeps running.

Changing pod resources historically required recreation, which made vertical autoscaling disruptive for workloads with expensive recovery. In-place resize changes that, with limits worth knowing.

Ingress solved routing by convention and annotation, which meant every controller invented its own. Gateway API's real contribution is not features, it is splitting one object into three so that platform and application teams stop editing the same file.

Clusters multiply for good reasons: regions, isolation, tenancy, blast radius. The problem is not creating them, it is that cluster seven diverges from cluster three and nobody knows until an upgrade behaves differently.

Sequence a compatible column migration, reconcile concurrent writes, and identify when returning to the old application starts requiring data repair.

Compare a shared-library patch across forty services, including affected builds, staged releases, runtime inventory and source-access boundaries.

Testing is a pipeline design problem in this role, not a QA one. The constraint is the question: you cannot run everything on every commit, so what you choose to defer is the answer.

Self-heal did exactly what it was configured to do. The response needs an authorized emergency path, a controlled reconciliation pause when required, and evidence that automation can resume safely.

Redeploying the old tag reverses code and nothing else. What separates staff-level answers is a per-change map of what rolls back in seconds, what rolls back in days, and what only moves forward.

Public fork PRs normally get a read-only token and no repository secrets. Privileged triggers, runner access and untrusted shell input can break that boundary.

A billion-row backfill is a deployable, throttled, resumable job with checkpoints and verification gates. Ad-hoc scripts against prod are how it becomes an incident.

Rings, SLI gates and Git-expressed promotion: sync waves order resources inside a cluster, but fleet sequencing is an automation you design on top.

Define the five delivery measures from explicit events, distinguish failure from rework, and expose uncertain attribution and small samples.

Build graphs identify affected targets only when their dependencies are complete. The ladder from build-everything to content-addressed caching, hermeticity audits, and where undeclared inputs rot correctness.

Separate blob integrity from action-result trust, isolate cache writers and investigate differences found by sampled cold rebuilds.

Runner migration questions hide two different exams: whether a PR job can steal your cloud credentials, and why the fleet idles at half price all night. Isolation first, economics second.

The code did not change, the data did. That inverts what triggers the pipeline, what the tests assert, and what a rollback means.

Deploying the application per pull request is a solved problem. Giving it data it can work against, and dependencies it can call, is where these projects stall and where the data-protection incident lives.

A token naming the right repository does not prove that the approved deployment workflow ran. Bind the caller, deployment context and called workflow into the cloud trust contract.

Defining chaos engineering is easy. Far fewer can say what they would break first, and almost none volunteer the abort condition, which is the thing that makes it a programme rather than an incident.

The observability failure that scales with your success. Cardinality is the cost driver in every metrics system, and the fix has to be structural or it comes back next quarter.

Alert fatigue is a design failure, not a discipline failure. The fix is a rule about what may page at all, and burn-rate alerting is how you implement it.

The instinct is to delete old logs. That is backwards: the waste lives in what you ingest, not how long you keep it. This answer walks the levers in the order that actually pays.

The pull model is the famous part and explains none of the operational pain. The storage engine is where Prometheus hurts in production: crash recovery, cold queries, disk death.

Anyone can divide bad requests by total. This answer shows the aggregation mistakes that quietly corrupt SLO dashboards, the rate-versus-increase traps that make budgets lie, and when subqueries cost too much.

Spending the budget is arithmetic; enforcing consequences is politics plus automation. The answer that lands names trigger thresholds, the deploy gate that blocks releases, and the dated escape valve.

Request-shaped SLIs quietly fail on data platforms. The strong answer defines freshness, completeness and deadline SLIs on consumer-visible artifacts, not job success rates.

Where linear models break: compounding growth, step functions, peak days and saturation cliffs. Forecast per constrained resource, backtest, subtract lead time to get the real order-by date.

Their green page, your red SLO. Separate their true failure rate from the amplification your own client adds, buffer at the boundary, then escalate with a client-side scorecard instead of feelings.

An SLA is a price list for failure signed before the failures. Bring the measured distribution, negotiate methodology and exclusions harder than the headline number, and phase targets with dates.

You have added a dependency with a latency distribution an order of magnitude wider than anything else in your stack, whose failure mode is a plausible wrong answer rather than an error.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

Tracing tells you which service and which span. It stops at the function boundary, and that is exactly where a CPU regression lives. Continuous profiling is the layer most teams do not have when they need it.

There is no status code and no caller waiting, so latency and error rate stop meaning what they meant. The signals that replace them are age, lag and completion, and they answer a question the request-response set never had to ask.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Intermittent DNS is the most-reported and least-understood Kubernetes failure. There are three well-known causes and each leaves a different fingerprint.

Both can be right, which is the whole point. Retransmissions mean a segment was not acknowledged, and the network is only one of the places that can happen.

The classic gap between what the backend measures and what the user experiences. A 504 is emitted by something in front of the backend, which is exactly the part your service dashboards do not cover.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

Established traffic keeps working while new connections die at random, which is exactly why this one confuses people. The table is full of flows that will not be needed again for days.

A two-second dependency should cost you latency, not three services. The mechanics are arithmetic: Little's law plus retry amplification. The fix is a budget, not a ban.

Same-node good, cross-node bad is not a random clue, it is nearly a diagnosis. One short decision tree separates overlay reachability, encapsulation blocks, MTU and policy.

An unchanged pipeline can fail because of environment state, external inputs or a latent artifact defect. Failures that start on an expiry boundary narrow the field to two suspects fast.

Load balancing makes a per-host fault look like a flaky library. The exact rejection string tells you which way the offending clock leans before you log into anything.

The kill was not random and it was not fair. Learn the scoring function the kernel actually uses, then bend it with oom_score_adj before your monitoring dies during the next leak.

The leaf is valid and the root is trusted, yet verification fails. The gap lives in the middle of the chain, and the exact verify code names it before you open a single config file.

Gradual drift produces no errors on the host that has it, only weirdness everywhere else. Sweep first, histogram second, repair third: the distribution shape names the culprit before you log into anything.

Duplicate-IP claims need evidence, and one command provides it. Then the real skill: telling a genuine two-owner clash apart from ARP flux, because the fixes could not be more different.

Every client fails at once, which rules out pods and points at shared PKI. Know which certificates share one expiry clock, and a controlled recovery that preserves cluster state.

Mixed resolution after a transfer is not propagation voodoo. Compare the parent delegation, authoritative answers and validation results. NXDOMAIN and SERVFAIL are clues; neither names a unique cause.

Split-brain writes are a fencing failure first and a data problem second. Freeze writes, read the divergence boundary from replication positions, and let the payment ledger arbitrate reality.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

A reported successful rebuild still leaves several places to investigate: a cache, a routing pointer, a chunk that survived deletion, or a document that never reached the pipeline at all.

The automation stopped because it could not decide, which is the correct behaviour and an incomplete design. Somebody has to decide, and the interesting question is what evidence that decision needs.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

A loop with no step cap has no natural ceiling, and runaway spending keeps accruing while you investigate. Stop the spend first, then find out what it was doing.

The disk is not full, the keyspace quota is. Defragmenting without disarming the alarm leaves the cluster read-only, and disarming without compacting puts it back within minutes.

Two controllers with different opinions and no shared hysteresis. Flat traffic with an oscillating node count means the loop is driven by something other than demand, and the candidates are few.

The heap is the part you configured and the smaller part of what the process uses. The kernel counts everything the cgroup touches, including the page cache the JVM never asked for.

Code rolled back and the database did not. The old version is now running against a schema it has never seen, and that is a worse incident than the one you were trying to end.

One partition has one consumer, by design. If the key distribution sends a third of the traffic to one partition, the parallelism you bought is unavailable to the work that needs it.

Workload authentication, traffic encryption and authorization need separate verification. Admission and RBAC constrain who can deploy a workload that bypasses those controls.

The single highest-value security change available to most pipelines, and the reason it works is worth understanding rather than configuring. It removes the secret rather than protecting it.

Three terms that get used interchangeably and answer three different questions. The one that makes them operational is the admission rule at the end: no provenance, no deploy.

Four tools that solve the same sentence and differ on one question: where does the plaintext end up? In a regulated environment that question decides the answer.

Everyone endorses least privilege; few can describe how they would measure over-permission across a fleet. The scoring answer names the evidence source, the tiered remediation, and the break-glass hatch that makes it survivable.

Signing proves what shipped, not what runs. The strong answer covers syscall-level detection, the signals that matter, tuning against alert fatigue, and the response path once a sensor fires.

The reframe that scores: many controls contain machine-checkable assertions; others require human evidence. Guardrails prevent, gates stop pipelines, drift detection watches live state, and evidence falls out as a byproduct.

Miners prove unauthorized workload execution, not a particular intrusion sequence. Parallel identity and network containment, backup control-plane protection, rebuild-don't-clean, and restore points older than patient zero.

Exceptions rot when they lack expiry dates, named owners and an aggregate view. Structured records where engineers work, auto-expiry with teeth, and a monthly portfolio review that treats growth as a control failure.

Most insider incidents are boring: a departing engineer copies a repo, a contractor keeps a token. Buy hygiene rather than surveillance, and protect the culture that makes people report their own mistakes.

Trace who can read ciphertext, who can decrypt it and who can administer the keys. Key rotation, rewrapping and data re-encryption solve different problems.

The classic napkin-math question. The number in the prompt is the question. Drawing the architecture before doing the arithmetic produces a design the physics forbids, and the arithmetic takes a minute.

The design round where the honest answer is mostly about data. Stateless traffic across regions is straightforward; the database is where active-active is either solved or pretended.

The migration question, and the trap is answering with a target architecture. What is being scored is sequencing: how you get there without a rewrite and without a release freeze.

The Indian e-commerce interview question, and autoscaling is the wrong first answer. A 10x spike arriving in seconds outruns any scaling loop, so the design has to absorb rather than scale.

Every candidate says replication and failover. The scoring answer names the replication mode, states the resulting data loss window, and explains how the cluster avoids two primaries.

Everyone names token bucket. The scored part is the rest: drift between nodes, the 429 contract with clients, and whether your quota service should fail open or closed.

The catalog is the easy half. The scored half is stampede dynamics: one hot key expiring can multiply origin load by a thousand in a second, and there are three distinct fixes.

Cron is the most-deployed scheduler on earth and the worst-behaved at scale. The scored content is everything crontab cannot do: exactly-once firing, thundering herds, and what happens when the scheduler itself dies.

Timeouts guarantee uncertainty, not failure: the first request may have succeeded after the client gave up. The idempotency key pattern is the industry answer; the scored details are storage, expiry and concurrency.

Outbound delivery looks like a queue and a loop until one subscriber hangs for sixty seconds and takes your workers with it. Isolation, retry horizons, and signatures are where the marks are.

Sharding buys write headroom and bills you in joins, foreign keys, and every future migration. The key choice is a query-pattern bet; the resharding choreography separates operators from diagram artists.

Every overloaded system dies the same way: queues turn load into latency, latency into timeouts, timeouts into retry storms. Tiered shedding, deadline propagation, and retry budgets are the design.

Everyone sizes a metrics system by samples per second and everyone is wrong. The constraint that actually kills these platforms is a single label added on a Friday.

Dual-writing from application code is the answer that fails quietly six months in. Change data capture, per-entity ordering, and a reindex you could run on a Tuesday are what interviewers score.

This is a machine identity problem wearing a secrets costume. Candidates who start at encryption miss that authentication of the caller and delivery mechanics are the real design space.

Listing the match job is table stakes. The scored material is the breaks taxonomy, the suspense account, and why every match must be replayable from immutable raw files.

Drawing an agent that polls a server is the easy third. The interview is about partial applies, fleet-versus-world state, and why rollback must be a pointer flip, never a reverse edit.

This is a platform-organization question disguised as infrastructure. Score comes from separating enforcement from ownership and explaining what stops the gateway becoming a smart ESB.

Trace a retry through the broker, database and external provider. The guarantee changes at each boundary, and a recorded intent does not prove a payment or email completed.

A disputed invoice needs a reproducible calculation. Retain raw usage events and versioned pricing so each line can be explained and corrected.

Queue time and build time are separate problems with separate fixes, and most teams spend a quarter optimising the one that is not hurting. Size the fleet from arrival rate, then attack the build with caching and graph awareness.

Comparing the canary against last week is how a Tuesday afternoon looks like a regression. Compare it against a baseline running the same traffic at the same moment, and decide with a threshold somebody can defend.

Most migrations are safe and a handful can take the site down by holding a lock for eight seconds. Build the tooling to tell those apart automatically, because a review culture cannot do it two hundred times a month.

The auditor does not require a meeting, they require evidence that changes are authorised, tested and reversible. Generate that evidence from the pipeline and the board stops being the bottleneck.

Turning on strict mTLS across two hundred services in one change breaks everything that talks to something outside the mesh. Permissive mode exists so the migration can be observed instead of guessed at.

DNS time-to-live is a suggestion and nineteen minutes is what believing it costs. The mechanism you choose decides your failover time, and the health check you choose decides whether it fires at all.

Erasure is a distributed transaction across systems that were never designed to participate in one, and backups make a literal delete impossible. The design that works is a deletion ledger plus crypto-shredding, not a fan-out of delete statements.

A single FIFO queue is a first-come-first-served promise to whoever submits fastest. Fairness needs per-tenant queues and a scheduler that picks between them, and priority without fairness just moves the problem.

A question about method, not about tactics. Candidates who open with a list of levers lose to candidates who open with one question about the bill itself.

A platform question that is really a product question. The design is the easy half; the answer that scores explains how you would know developers chose it rather than were forced onto it.

Four purchase instruments, one fleet, and a finance team asking why you did not commit to everything at maximum discount. The scoring rubric rewards the arithmetic of being wrong, not the discount table.

Most of your rows are read rarely and cost the same as the ones read constantly. Tiering pays for itself only when access patterns are honest, which is why the audit comes before the architecture.

Token spend has no natural ceiling. Every retry, every retrieved document, every chain step multiplies, and nothing in the system tells anyone until the invoice arrives.

The discount is real and so is the option you are selling. The question is what your usage looks like in year three, and the honest answer for most teams is that they do not know.

Announcing a deadline is not a migration plan. The teams that miss it are the ones who were never going to read the announcement.

An internal platform with no stated objective still has one. Every team guesses it, they all guess differently, and the guesses only surface during an outage.

Each team is behaving rationally. The cost of migrating is theirs, immediate and certain; the benefit is later and uncertain. That asymmetry does not resolve by being explained again.

You cannot run the counterfactual. What survives scrutiny is a comparison design that narrows the alternatives, not a number that claims to settle it.

Saying yes without a contribution contract produces features the platform team did not design and now supports forever, on call, at 3am.

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

The driver, the container toolkit, the CUDA runtime inside the image and the framework build all have to agree. Upgrade the wrong one first and every pod on the node fails to start with an error that names none of them.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.

Both workloads are legitimate and they want opposite things from the same GPU. Combine tenant rate limits with fair admission and measured capacity reservations; separate fleets are an option when shared scheduling cannot meet the targets.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Asking nicely in the prompt gets you most of the way and fails on the tail, which is where the automation breaks. Constrained decoding can guarantee the shape of completed output for supported schemas. Interrupted generations still need a failure path, and compilation has a latency cost.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

Scale to zero is correct for some workloads and an outage generator for others, and the deciding number is what the first request after idle experiences. Cold start turns an availability promise into a queue.

A model provider is a single dependency with a shared failure domain across every customer they have. Treating it like a database you cannot fail over from is how a product goes down for four hours.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

Prompts contain whatever the user typed, which in a support product is names, account numbers and medical detail. Full-text logging is a data-protection decision, not a debugging preference.

The training part is a job. Everything around it is the pipeline: data snapshots that are immutable, an evaluation gate with a threshold set in advance, and adapters versioned so a bad one is a routing change away from gone.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Code and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.

A model is an artifact with a lineage, an evaluation record and an owner, or it is a file somebody copied to a bucket. The registry exists so that the second one cannot reach production.

Before blaming the model, check whether it is being fed the same thing it was trained on. Most of the gap is a pipeline difference, and pipeline differences are a platform problem with a platform fix.

The database part is easy. The operational problems are reindexing when the embedding model changes, recall degrading silently as filters get stricter, and memory growing faster than anyone forecast.

A release-automation design question with a specific trust boundary. The novel risk is prompt injection: untrusted text reaching a privileged agent, where the text is the instruction and your usual input validation does not apply.

The valuable part is the first ten minutes of context gathering, which is tedious and mechanical. The dangerous part is anything that changes production while a human is still working out what is wrong.

Generated infrastructure code is plausible, fluent and confidently wrong in specific ways. The volume goes up, the reviewer attention per line goes down, and the controls that used to be optional stop being optional.

Every team standing up its own tool server produces a permission surface nobody can enumerate. The gateway exists so that connecting an agent to a system is a reviewed act rather than a config file in somebody's repository.

Per-token pricing looks expensive until you compute what an idle GPU costs at 3am. Break-even depends on workload volume and duty cycle, measured throughput, and the engineering cost on both sides.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

Nobody is waiting, which changes every constraint. Latency stops mattering, throughput per dollar becomes the only metric, and the design converges on something that looks like a data pipeline rather than a service.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

GPUs are the default for reasons that hold for large generative models and not for everything else. A small classifier on CPU can be cheaper, more available and easier to operate, and knowing where the crossover sits is worth more than knowing the newest card.

A model artifact is code that runs in your cluster with GPU access. Most of the supply-chain controls you already run for containers apply, and the file format adds one of its own.

A mandatory gateway can centralize limits and filtering, while applications authorize business actions. Neither a prompt nor a PII classifier guarantees that sensitive data stays private.

Most of the obligation lands on the product, but three parts land on infrastructure: an inventory of what is deployed, evidence of what it did, and the ability to turn it off. Those are platform capabilities and they take months to retrofit.

Coordinate a major incident through explicit roles, bounded investigations, safe recovery decisions and a handoff that preserves current state.

Retire an internal tool through verified user needs, compatibility evidence and an owned cutover, while treating its maintainer as a source of context rather than a barrier.

Write an accurate incident review that examines the triggering action and contributing conditions, with corrective actions whose effectiveness can be checked.

Defend a change with irreversible effects by separating code rollback, data recovery and external correction, with explicit recovery objectives and tested limits.

Plan an on-call transition around service readiness, responder capacity and authority, with rehearsed escalation and evidence that the operating burden is sustainable.

Assess a requested control bypass against its purpose and approval rules, then propose a permitted delivery path and document the actual decision.

Decide whether to continue a migration using remaining costs and current benefits, then deliberately operate or unwind the partial state.

Define the shared operating requirements before standardizing delivery, then test representative migrations and keep supported exceptions explicit.

Assess a signed vendor choice against the actual requirements and contract, then support a bounded implementation or raise material new evidence through its owners.