batching

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.