fairness

A single FIFO queue is a first-come-first-served promise to whoever submits fastest. Fairness needs per-tenant queues and a scheduler that picks between them, and priority without fairness just moves the problem.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

Both workloads are legitimate and they want opposite things from the same GPU. Combine tenant rate limits with fair admission and measured capacity reservations; separate fleets are an option when shared scheduling cannot meet the targets.

Keep incident coverage working while investigating missed notifications, with private conversation, tested escalation and appropriate management support.