p99 spikes, probes time out, CPU utilisation reads 30 percent and nothing is OOMKilled. cpu.stat shows nr_throttled climbing. Explain it.

TL;DR: A CPU limit commonly enforces quota per 100ms period; inspect the configured period, and a multi-threaded runtime burns the whole period's budget in a few milliseconds of parallel work, then freezes until the next period. Utilisation graphs average over seconds, so a mean-usage panel can hide those stalls. Read nr_throttled against nr_periods in the container's cpu.stat.

How to approach it

Name the mechanism before proposing fixes: this is CFS bandwidth control behaving as designed, not a scheduling bug. Then ask for two numbers, throttled periods over total periods, and the thread count of the process. Both decide which fix applies.

A strong answer

The kernel accounts CPU bandwidth in fixed periods, 100ms by default. A limit of one core (1000m) grants 100ms of runtime per period. The subtlety is that the budget is shared across every thread in the cgroup. A JVM or Go service with sixteen runnable threads spends that budget sixteen times faster: 100 / 16 = 6.25 milliseconds of wall clock, after which the quota is gone and the remaining 100 − 6.25 ≈ 94 milliseconds of the period are spent frozen. The ratio of running to waiting is roughly 1 to 15, which is the entire explanation for what the dashboards show next. In this simplified burst, the cgroup waits for the next period to receive another budget.

What does that look like from outside? If such bursts occur only in some periods, average CPU can remain below the limit while those bursts incur throttling. p99 is catastrophic, because any request straddling a throttle eats tens to hundreds of milliseconds of frozen time. Liveness and readiness probes are executed by the kubelet against a process that may be frozen, so they time out too, and if you also run a restart loop you have converted a latency problem into an availability one.

Confirm it from the cgroup rather than from inference:

# cgroup v2, inside the container's mount or via the node
cat /sys/fs/cgroup/cpu.stat
# nr_periods 48211 nr_throttled 9123 throttled_usec 771234000

Compute deltas of throttled periods and total periods over the incident window, then correlate them with latency and throttled duration. No fixed percentage proves causation or makes throttling harmless. The reason dashboards miss it is sampling: Prometheus scrapes at 15 to 60 second intervals and averages, and the freeze windows live inside individual 100ms periods.

The fix has three honest options, and the interesting part is the trade-off.

Remove the CPU limit, keep the request. Requests guide scheduling and CPU weights under contention; they do not cap aggregate runnable demand. Limits on CPU mostly buy protection from noisy neighbours that bin-packing should handle, and they charge you tail latency for it. This is my default for latency-sensitive services where the platform tolerates it.

Keep the limit and align parallelism to it. Check the runtime version and actual effective parallelism. Go 1.25 introduced container-aware GOMAXPROCS defaults, and modern JVMs can detect cgroups; old versions, overrides and compatibility settings can change that behavior. Set GOMAXPROCS (automaxprocs does this automatically), -XX:ActiveProcessorCount on the JVM, or worker counts explicitly, so four threads share a one-core quota instead of sixteen. Throttling shrinks because demand stops exceeding supply inside every period.

Raise the quota. Legitimate when the service genuinely needs more CPU and the graph was lying about headroom. It is the least interesting answer and the one candidates reach for first.

I reverse the default when nodes are heavily overcommitted with untrusted neighbours, as in mixed tenancy: there the limit is doing real isolation work, so align threads and keep it.

Read the dashboard denominator and the active interval

A graph showing 30% CPU is ambiguous until you know its denominator. It might mean 30% of the limit, 30% of a request, or 30% of the node's cores. Read cpu.max on cgroup v2 to confirm the configured quota and period, then compare usage and throttling deltas over the same interval as slow requests.

The sixteen-thread example assumes sixteen threads can run simultaneously on available CPUs. If that burst repeats every period and consumes the full quota each time, average usage approaches the one-core limit; it does not explain a 30%-of-limit graph by itself. Low average usage with throttling requires intermittent bursts, a different denominator or another relevant accounting detail.

Test a bounded parallelism change on one replica at comparable demand. Observe completed requests per second, pool wait and tail latency as well as throttling. Fewer throttled periods with a much longer application queue is not an improvement. Keep the original configuration available and compare the complete request path.

What interviewers probe next

"Why doesn't the CPU metric show saturation?" Because usage is averaged across the scrape interval and across cores, while throttling happens within single 100ms periods. Sub-period bursts are invisible at dashboard resolution.

"Does memory behave the same way?" No. Memory has no CPU-style quota period. The kernel can reclaim or throttle memory allocations before an OOM kill; cgroup-v2 memory.high and memory.max have different roles. That asymmetry is why CPU limits are widely considered more harmful than useful and memory limits are not negotiable.

"Where else does this same mechanism bite?" Anywhere CFS quotas apply: systemd services with CPUQuota (CPUAccounting only measures usage), CI runners, serverless sandboxes. Same math everywhere.

Common mistakes

Reading nr_throttled alone without nr_periods. Compare counter deltas, duration and affected request latency. A small fraction can still matter to a strict tail-latency target.

Raising replicas instead of fixing parallelism. Horizontal scaling spreads load but every pod still freezes for 94ms per bad period, so measure whether distributing demand actually reduces quota exhaustion before treating replica count as a fix.

Blaming the network because probe timeouts look like connection failures. The application may lack CPU time to answer the probe; check whether the timeouts cluster at the same cadence as the throttle period before leaving the box.

References

Go 1.25 runtime changes. systemd resource controls.