monitoring

Heartbeats, conditions and taints are the only vocabulary the control plane understands. NPD translates journald and kernel ring messages into that vocabulary; custom plugins carry your hardware's dialect.

Prometheus asks your service for numbers on a schedule rather than waiting to be told. That one design choice explains service discovery, exporters, staleness and why short-lived jobs are the awkward case.

Somebody silenced it during an incident nine days ago to stop the noise, and nothing ever expired the silence. The monitoring was working perfectly and nobody was listening.

Most insider incidents are boring: a departing engineer copies a repo, a contractor keeps a token. Buy hygiene rather than surveillance, and protect the culture that makes people report their own mistakes.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

Before blaming the model, check whether it is being fed the same thing it was trained on. Most of the gap is a pipeline difference, and pipeline differences are a platform problem with a platform fix.