TL;DR: Restore service first, understand second. Find the biggest consumer, reclaim safely, and confirm the space actually returned, because a deleted file held open by a running process frees nothing. Check block and inode usage before choosing the cleanup, since both can produce ENOSPC and need different remedies.
How to approach it
Lead with the order of operations, because that is what is being scored at 3am. Then show you know the open-file-handle trap and the inode case, which are the two things that make this more than a cleanup exercise.
A strong answer
First, confirm scope and buy time. Is it one node or many, and is anything actually failing? A full disk usually presents as write failures, a database refusing connections, or a kubelet evicting pods under disk pressure.
Then find it, cheaply:
df -h # filesystem block usage
df -i # inodes: 100% here with free space above is the trap
du -x --max-depth=1 / | sort -rh | head # -x stays on one filesystem
du -x matters. Without it you wander into mounted volumes and network filesystems and waste minutes at exactly the wrong time.
The usual culprits in order of likelihood: application logs that never rotated, container image layers and stopped containers, a core dump from a crash loop, and an unbounded temp directory. On a Kubernetes node, container logs under /var/log/pods and the image cache are the two big ones, and crictl or the kubelet's garbage collection settings are the right lever rather than deleting files by hand.
Now the trap that catches people. Deleting a file that a process still holds open does not return the space. The directory entry is gone and the inode stays allocated until the last file descriptor closes. So du shows the space freed and df still shows the disk full. Confirm with:
lsof +L1 # files with zero links still held open
For an unlinked log, use the application's documented reopen signal or a controlled restart so it closes the old descriptor. Truncating the old pathname will create a new file and leave the unlinked file allocated. Truncation can reclaim a still-linked, disposable log, but destroys its contents and may interact badly with non-append writers; preserve needed incident evidence and prefer supported rotation. Never apply log cleanup techniques to database files.
The inode case deserves its own mention: a filesystem can have gigabytes free and still fail every write because it is out of inodes, caused by millions of tiny files such as session files or an unbounded mail queue. df -i distinguishes it in one command, and remove safe-to-delete small files and fix their retention. Capacity options depend on the filesystem: growing ext4 adds inodes in new block groups at its configured density, while XFS allocates inodes dynamically subject to its limits. Check the filesystem and growth procedure before ruling out expansion.
Once service is restored, the follow-up is prevention rather than cleanup: log rotation with a size cap, disk usage alerting with enough headroom to act, image garbage collection thresholds on nodes, and shipping logs off-node with bounded local buffering, so a destination or shipper failure cannot consume unlimited disk.
Choose recovery by mount and data ownership
Suppose /var is full but /data has ample space. Run df for the failing application's actual path, then inspect that filesystem. Freeing a large file on /data will not help a write to /var. Check whether the failure is block capacity, inode capacity or a quota applying to the process.
If a disposable log is the largest consumer, identify its writer and supported rotation mechanism. Preserve a bounded incident sample, reopen the log through the application and verify df changes. For database files, involve the database recovery procedure; deleting a WAL segment or storage-engine file because it is large can make the outage unrecoverable.
Measure the growth rate after reclaiming space. Releasing 5 GiB while a failed shipper accumulates 1 GiB per minute buys only five minutes. Restore shipping or cap the producer's retained logs, then confirm application writes succeed and headroom remains stable. Cleanup is complete when the failure stops recurring, not when a removal command exits successfully.
What interviewers probe next
"Why does df disagree with du?" Either open file handles as above, or you are looking at different mount points. Both are worth naming.
"How do you alert on this usefully?" On rate of growth as well as absolute level. A stable level and rapid growth need different responses; retain enough absolute headroom for bursts and recovery operations in both cases.
"What is reserved space?" Ext filesystems reserve around five percent for root by default, which is why the disk fills for the application before it fills for you, and why you can still log in and fix it.
Common mistakes
Diagnosing at length while the service is down. Reclaim first.
Deleting an open log file and reporting it fixed without checking df.
Missing the inode case entirely, which makes the whole answer look confidently wrong when the interviewer reveals free space.