Your distributed training run died six hours into an epoch on preemptible GPUs. What happens next, and how do you design for it?

A training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.