NVIDIA SRE / Production Engineer interview questions

The swap-used graph is the least informative memory metric on the box. Here is what swappiness actually weights, which numbers to watch instead, and the policy worth defending.

SIGKILL writes no stack trace and the application cannot defend itself. The verdict is already sitting in the kernel ring buffer, and reading it precisely is the skill being priced.

One CPU doing all the packet work is a topology failure, not a tuning mystery. RSS queues, MSI-X vectors, local_cpulist and irqbalance: the whole chain from symptom to pin.

Most throughput tests measure the tester, not the network. Single stream versus parallel, slow start, sender-side retransmits, and the CPU ceiling: how to produce a number you can act on.

Guaranteed capacity and discounted capacity are different products. Peak season needs both, arranged in layers and verified with a game day before the traffic arrives.

General-purpose IaC earns its complexity where application engineers own infrastructure and benefit from typed components and unit tests. It disappoints when the pitch was freedom: the engine still builds a graph, still keeps state, and still forbids most of the clever runtime logic people expect to write.

Trace sandbox setup, address allocation and a cross-node packet, then distinguish pod reachability from Service translation.

Namespaces partition resources, not trust. This question finds the people who know which sysctls are namespaced, which ones change every tenant at once, and when only a second kernel will do.

Multiple schedulers are supported, rarely understood, and dangerous in a specific way. The strong answer names the deadlock that gang scheduling fixes and the capacity race it can open.

Dockershim is gone and the CRI contract decides what a runtime must do. The senior-level part is the shim model: a supervisor process that owns exit codes and stdio so the daemon can die without taking workloads with it.

Two cleanup mechanisms hide behind one taint: image garbage collection removes unreferenced images by LRU at threshold crossings, while eviction kills pods. Candidates who conflate them fail this question quietly.

Driver, container toolkit, device plugin, DCGM and discovery must version-match on every node. The operator reconciles that ladder from one ClusterPolicy CR; MIG, upgrades and host kernels are where it bites.

Kubernetes does not see GPUs, it sees an extended resource that something has to advertise. When that advertisement is missing or wrong, the scheduler is behaving correctly and the node is lying.

Base layers are dependencies: digest-pin them, let bots bump them weekly, and let CVE disclosures trigger scoped rebuilds through the normal pipeline. Days, not quarters.

Separate blob integrity from action-result trust, isolate cache writers and investigate differences found by sampled cold rebuilds.

The kernel watches syscalls, sockets and short-lived processes that userspace agents polling every few seconds never meet. Know what eBPF adds, what it costs, and the security conversation it starts.

NFS waits can stall shutdown and inflate load average while CPUs stay idle. Check the blocked kernel path before deciding whether signals or mount recovery can release it.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

The driver, the container toolkit, the CUDA runtime inside the image and the framework build all have to agree. Upgrade the wrong one first and every pod on the node fails to start with an error that names none of them.

At sixteen GPUs and above, the network is the computer. Collectives run at the speed of the worst link in the group, and placement decides which link that is.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

A training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Code and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

The question that separates people who have debugged a hung rollout from people who have read about one. The interesting half is not the signal, it is PID 1.

Ordering and retransmission are what TCP adds, and both of them cost time. Naming something that deliberately gives them up, and saying why, is what turns a textbook answer into a real one.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

A staple of every networking round, and the follow-up is always about TLS termination. Knowing which layer sees what decides most of the answer.

The oldest question in systems interviewing and still the best, because it is unbounded: every layer you can describe is another layer you could debug.

Five stages, and which one you are stuck in tells you what to fix. A machine that reaches a GRUB prompt has a different problem from one that hangs waiting on a filesystem that no longer exists.

The classic ephemeral port exhaustion question. TIME_WAIT is a familiar phrase and the useful facts are narrower: which side accumulates it, and why the popular fix is dangerous.