gpu

Kubernetes does not see GPUs, it sees an extended resource that something has to advertise. When that advertisement is missing or wrong, the scheduler is behaving correctly and the node is lying.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

A training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.

The training part is a job. Everything around it is the pipeline: data snapshots that are immutable, an evaluation gate with a threshold set in advance, and adapters versioned so a bad one is a routing change away from gone.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.