retention

Volatile by default on many distros, size-capped by silent policy, binary on disk. Know the storage modes, the vacuum knobs, and how journald feeds central logging.

A registry grows because every commit mints digests nobody pulls again. Reference-aware retention keeps rollback targets and audit chains while expiring the debris.

The instinct is to delete old logs. That is backwards: the waste lives in what you ingest, not how long you keep it. This answer walks the levers in the order that actually pays.

The pull model is the famous part and explains none of the operational pain. The storage engine is where Prometheus hurts in production: crash recovery, cold queries, disk death.

Everyone sizes a metrics system by samples per second and everyone is wrong. The constraint that actually kills these platforms is a single label added on a Friday.

The hard parts are not search. They are deciding what to keep before you pay to store it, isolating one team's query from everyone else's, and making the cost visible to the people generating it.