incident

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

A reported successful rebuild still leaves several places to investigate: a cache, a routing pointer, a chunk that survived deletion, or a document that never reached the pipeline at all.

The automation stopped because it could not decide, which is the correct behaviour and an incomplete design. Somebody has to decide, and the interesting question is what evidence that decision needs.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

A loop with no step cap has no natural ceiling, and runaway spending keeps accruing while you investigate. Stop the spend first, then find out what it was doing.

An incident answer needs your actual decisions, the evidence behind them and the effect on users. A wrong turn belongs in the story only if it happened.