TL;DR: Model the fleet as distinct pools with distinct resource names rather than as one pool of interchangeable GPUs, so a workload's requirements are expressed in the request instead of implied by hope. Then decide per workload class: training pins to one generation because collectives run at the speed of the slowest member, while inference tolerates mixing if you route by capability and account for throughput differences in capacity planning.
How to approach it
Say up front that heterogeneity is permanent, because the answer changes if you treat it as debt to be paid off. Then split by workload class, since training and inference have opposite tolerances for it.
A strong answer
Why one pool fails. The device plugin advertises nvidia.com/gpu from every node regardless of what the card is. A job needing 80GB of device memory can be scheduled onto a 48GB card and fails at load with an out-of-memory error rather than staying Pending, which is a worse outcome because it consumes a scheduling cycle and looks like an application bug. The traditional workaround is labels from GPU Feature Discovery plus node affinity in every manifest, which works and pushes scheduling logic into workload specs where it drifts.
Separate pools, separate names. Give each generation its own node pool with its own taint, and where the platform supports it, advertise distinct extended resource names so a request states what it needs. Kueue resource flavors express this cleanly: one ClusterQueue can list flavors in preference order with quota per flavor. Workload node selectors constrain eligibility; flavor ordering and fungibility policies are configured on the ClusterQueue, not an arbitrary per-job preference-list API. See Kueue ClusterQueue flavor selection.
Training pins, and the reason is arithmetic, so here it is. A data-parallel job synchronises every step, which means every rank waits at the all-reduce for the slowest of them.
Take eight GPUs, seven current and one 40 percent slower, so the odd card completes a step in 1 / 0.6 ≈ 1.67 times the time the others need. Every step runs at that pace, so each fast GPU sits idle for 40 percent of every step and the group wastes 7 × 0.4 = 2.8 GPU-equivalents continuously. Job throughput is 8 × 0.6 = 4.8 units against the 8 it would have with a uniform group.
Now the result that settles the design question. Drop the slow card and run seven: throughput is 7 units, against 4.8 with the slow card included. Adding the slower GPU made the job about 31 percent slower than leaving it out, so the mixed group is worse than having less hardware, and no configuration recovers it because the loss is in the synchronisation barrier rather than in any tunable. There is no configuration that fixes this. Mixed-generation training is only reasonable when the parallelism strategy deliberately assigns different work per rank, which is rare and hard.
Inference mixes well if you plan capacity properly. Replicas are independent, so a slower card simply serves fewer requests. Two things follow. First, route by capability: a model that needs FP8 hardware support cannot go to a generation without it, and a model that needs 70GB cannot go to a 48GB card. Second, capacity is not replica count, and the gap is large enough to break a scaling policy. Suppose the current pool serves 2,400 tokens per second per replica and the older one 900. A fleet target of 20 replicas means anywhere between 20 × 900 = 18,000 and 20 × 2,400 = 48,000 tokens per second depending on where those replicas land, a factor of 2.7 on the same replica count. Size each pool from its measured tokens per second and let the autoscaler target throughput rather than replicas, or the fleet under-provisions by that factor on the day traffic lands on the older pool.
Cost accounting has to follow the hardware. Charging one blended GPU-hour rate across generations means a team on old cards subsidises a team on new ones, and neither can see it. Rate per pool, reported per team.
What interviewers probe next
"How does DRA change this?" It replaces the countable resource with a claim describing attributes, so memory size, model and topology become expressible in the request rather than encoded in labels. That is precisely this problem, which is why the feature exists.
"What do you do with the oldest generation?" Give it the work that tolerates it: batch inference, evaluation runs, development, CI. Retiring it early wastes capacity you already own, and letting it into the training pool wastes capacity you just bought.
"How do you stop teams all requesting the newest?" Price it, or queue it. If the newest pool is free and unlimited, every job requests it and the older pools sit idle while the queue grows.
Common mistakes
One pool, one resource name, and node affinity copied between manifests until nobody remembers which constraints still matter for which job.
Mixed-generation distributed training, which silently runs at the slowest card's speed and gets diagnosed as a code regression.
Autoscaling on replica count across a mixed fleet, which produces a capacity number that is wrong by whatever the generation mix happens to be that hour.