TL;DR: Kubernetes schedules pod by pod, so a job needing sixteen GPUs can acquire twelve, wait for four more, and hold them while a second job does the same thing in reverse. Neither starts and neither releases. Use whole-workload quota admission, placement checks and a readiness timeout that releases resources when the gang cannot start. Kueue quota reservation alone cannot guarantee that pods will fit on physical nodes.
How to approach it
Name the deadlock before proposing tooling, because every option that follows is a different way of preventing the same thing, and without the mechanism they are indistinguishable logos. Then separate the three problems this creates: admission, fairness, and placement quality.
A strong answer
The deadlock. Distributed training is a gang: all ranks must be running before rank zero can complete the first collective. The default scheduler has no concept of a gang, so it binds whatever pods it can. Two jobs each wanting sixteen GPUs on a twenty-GPU cluster can interleave placement, acquire ten each, and both wait. GPUs are allocated, utilisation reports zero, and nothing progresses until a human kills one job. Check pod events, resource requests, affinity and device health before concluding this is the cause.
Whole-workload quota admission reduces this risk. Kueue reserves quota before unsuspending a supported job. Aggregate quota does not prove that each pod fits: eight free GPUs split between two nodes cannot host one eight-GPU pod. Enable topology-aware scheduling to check placement against node capacity and topology domains. Configure waitForPodsReady to evict and requeue a workload that misses its readiness timeout; blockAdmission can hold subsequent admissions until the preceding workload is ready. Node failures and provisioning delays still need handling. See Kueue all-or-nothing scheduling. Volcano provides scheduler-level gang support through PodGroups; set the minimum membership to the ranks the training job actually needs.
Fairness needs a quota model, not a priority number. ClusterQueues grouped into a cohort let teams borrow idle capacity from each other and have it reclaimed when the lender returns. Without borrowing, quota means stranded GPUs at every hour when a team is not training. Without reclaim, "borrowing" means the loudest team keeps the cluster. Preemption policy is where the real conversation happens: a research job with checkpointing every ten minutes is safe to preempt, a job that checkpoints hourly is expensive to preempt, and a job with no checkpointing at all should not be admitted to a preemptible queue in the first place.
Placement quality is separate from admission. Sixteen GPUs across sixteen nodes may satisfy rank count but perform poorly if collectives cross a slow or congested link. Placement wants topology awareness: prefer whole nodes, prefer nodes inside one NVLink or rail-optimised network domain, and express that with topology constraints rather than hope. On cloud, that usually means placement groups or capacity blocks; the scheduler cannot invent locality the underlying network does not have.
| Symptom | Cause | Fix |
|---|---|---|
| Pending with free GPUs | Partial placement or fragmentation | Inspect events; quota, topology checks and readiness timeout |
| Idle quota, queued jobs | No borrowing across queues | Cohorts with reclaim |
| Slow epochs, correct GPU count | Fragmented placement | Topology-aware assignment |
| Repeated restarts from rank 3 | One unhealthy node in the gang | Health gating before admission |
What interviewers probe next
"How long should a job wait?" Queue time is an SLO like any other. Publish it per queue, alert when the p90 breaks, and treat a persistent breach as a capacity decision rather than a scheduling bug.
"What happens when one rank dies at hour six?" The gang restarts from the last checkpoint, which is why checkpoint interval is a scheduling parameter and not a training detail. Elastic training helps for some frameworks and is not free.
"Preempt the big job or the small one?" Estimate lost GPU-hours and restart cost. Kueue selects victims using configured priority and quota/fair-sharing policies; it does not automatically inspect checkpoint age. A checkpoint-aware victim selector requires additional integration. Align job priorities and checkpoint contracts with the supported policy first.
Common mistakes
Installing Volcano or Kueue and never configuring quota, which converts a scheduling deadlock into a first-come-first-served queue with extra YAML.
Assuming a pod that is Running is doing work. In a gang, every pod runs and none progresses until the last one joins.
Ignoring node health in admission. Admitting a gang onto a node with a degraded NVLink produces a job that runs at a third of expected speed and looks like a code regression.