TL;DR: Classify requests at the gateway into priority tiers, enforce per-tenant token budgets so no single caller can occupy the batch, and let the serving engine admit by priority with the interactive tier reserved a share of the KV cache. Separate fleets are the fallback when the tiers have different hardware needs or a contractual isolation requirement.
How to approach it
Distinguish rate control from scheduling: rate limits bound a tenant's burst or sustained consumption, while fair admission decides which eligible request runs next. Separate fleets add headroom costs, but do not automatically double them. Then build the answer from where the contention actually is, which is admission into the batch and occupancy of the cache.
A strong answer
Name the contention precisely. The batch job is not stealing CPU. It is occupying KV cache slots and taking places in the running batch, so an interactive request arrives and waits for admission. That is why latency degrades rather than throughput, and it is why the fix belongs at admission rather than in scheduling policy elsewhere.
Classify at the gateway. Every request carries a priority derived from the endpoint and the tenant, not from anything the caller sets. Interactive chat is high, asynchronous summarisation is low, an internal evaluation run is lowest. Three tiers is usually enough, and more than four becomes a policy nobody can reason about.
Enforce a reservation, not just an ordering. Pure priority ordering starves the low tier whenever the high tier is busy, which turns a latency problem into a batch job that never finishes. This is a proposed scheduling policy, not a stock vLLM guarantee of per-tier KV reservations. Verify engine support or implement and test a controller before promising it. Reserve a floor for each tier: the interactive tier is guaranteed a share of cache and admission slots, and the low tier is guaranteed a smaller share it can always use. Above the floors, capacity goes to whoever is waiting, by priority.
Per-tenant budgets stop the specific failure in the question. Priority tiers handle workload classes; they do not stop one tenant in the low tier consuming the entire low-tier allocation. A token budget per tenant per window, enforced at the gateway, is what makes the batch tier fair internally. Without it the tier is first-come-first-served and the tenant with the most aggressive client wins.
Preemption is available and it is expensive. Some engines can evict a running low-priority sequence to admit a high-priority one, recomputing the evicted work later. That converts wasted latency into wasted compute, which is the right trade when the interactive SLO is contractual and the wrong one when the batch job is on a deadline. Say which case you are in.
When to separate fleets. Three conditions justify it: the tiers need different hardware (long-context batch wanting large memory, interactive wanting lowest latency), a customer has a contractual isolation requirement, or the batch tier is large enough that its own pool runs at high utilisation. Otherwise the shared fleet with admission control wins on cost, because two pools each need their own headroom.
What interviewers probe next
"How do you set the reservation sizes?" From the interactive tier's peak concurrency plus headroom, measured. The batch floor is then what remains, and if that is too small the answer is more capacity rather than a smaller interactive reservation.
"What do you tell the batch customer?" That their work runs at lower priority with a guaranteed floor and a published expectation, ideally with a cheaper price attached. Priority tiers work best when they are a product decision the customer chose, not a policy imposed silently.
"How do you detect this happening?" Queue time by tier, as a first-class metric. Aggregate queue time hides it completely, because the batch tier's long waits are correct.
Common mistakes
Using only a fixed request-rate limit when token lengths vary, or using priority without tenant budgets and fair scheduling.
Pure priority with no floor, which starves the low tier and produces a batch job that never completes.
Trusting a priority field supplied by the caller, which means every client marks itself urgent within a week.