storage

A routine cloud operation with three layers that can quietly disagree. The strong answer is an ordered procedure you could run during business hours, without a reboot.

Copy-on-write freezes a point-in-time view while the database keeps writing. Fat snapshots overflow into invalid backups and thin pools exhaust into frozen volumes: size both deliberately.

S3 access failures are rarely about the bucket policy alone. The caller, object ownership and network path determine which authorization checks apply. Start with the failed request and trace changes across those boundaries.

The writable layer is part of the container, not the image, and it dies with the container. Everything about volumes follows from knowing which of those two a file is in.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

Almost always a finalizer, and the fast fix is the one that loses data. The safe answer explains what the finalizer is protecting against before removing it.

Emergency mode after a storage change is usually one line in fstab. The order of operations is what decides the outcome here, along with whether the recovery preserves data-mount dependencies rather than booting past them.

NFS waits can stall shutdown and inflate load average while CPUs stay idle. Check the blocked kernel path before deciding whether signals or mount recovery can release it.

The volume is in one availability zone and the scheduler cannot move it. Spare capacity in the other two zones is not spare capacity for this pod, and no amount of scaling helps.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.