DevOpsInterviews logo
← Linux, Networking & Scripting / 18
hard★ EssentialNetflixUberLinkedIn

Your service is collecting thousands of CLOSE_WAIT sockets. What does that state mean and how do you hunt the cause?

A growing backlog of sockets waiting for the application to close its side usually points at lifecycle handling. Here is the hunt from ss output to the pool timeout that caused it.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: CLOSE_WAIT means the peer sent FIN, the kernel acknowledged it, and your application has not called close(). Nothing ages the state out, so every leaked socket holds a file descriptor until the process dies. The fix lives in application code and pool timeouts, never in sysctls.

How to approach it

Place the state precisely in the close sequence first, then turn the question into a hunt: which process owns them, which peer is closing, and what event correlates with the spike. Interviewers at this level are checking whether you debug leaks or restart around them.

A strong answer

When a peer closes, its FIN arrives, the kernel ACKs it, and the socket moves to CLOSE_WAIT. The kernel's work is done; the protocol now waits for your application to notice EOF on read and call close(). No kernel timer sweeps this state. tcp_fin_timeout governs orphaned FIN_WAIT_2 sockets, not CLOSE_WAIT, and tuning it for this problem is the most common wrong answer in circulation.

Why CLOSE_WAIT sockets accumulate and no timer removes them THE KERNEL'S HALF YOUR HALF 1 The peer closes and sends its FIN 2 Our kernel ACKs it as the protocol requires 3 Socket enters CLOSE_WAIT and the kernel is done 4 Now it waits for you read returns end of file 5 The app should close() and this one does not 6 No timer sweeps it there is no timer for this 7 The descriptor is held along with its buffers 8 Until the limit is reached too many open files The whole diagnosis in one line. A growing CLOSE_WAIT count is not a network problem and not a tuning problem: it names a code path that stopped reading and never closed. tcp_fin_timeout governs orphaned FIN_WAIT_2 sockets, not this state, and tuning it for this problem is the most common wrong answer in circulation.
StateWho reached itMeaning
FIN_WAIT_1 / FIN_WAIT_2local side closedawaiting peer ACK / awaiting peer FIN
CLOSE_WAITpeer closed firstour application owes a close()
LAST_ACKpassive closer sent FINawaiting final ACK
TIME_WAITactive closer, after full closehealthy 2MSL dwell

Leaks have three usual shapes. Connection pools whose idle lifetime exceeds an upstream idle timeout: an ALB default of 60 seconds sends FIN to pooled connections that the client still believes are alive, and each one sits in CLOSE_WAIT until reuse fails. Missing cleanup paths: a response body or stream closed only on the happy path, so exceptions leak one fd per failure. Half-close mishandling: reading EOF but never closing the write half, legal in TCP but fatal when nobody ever writes.

The cost is one file descriptor per leaked socket, and the time-to-failure is worth computing because it sets how urgent this is. Soft limits commonly ship at 1024. With perhaps 50 descriptors in legitimate use there are about 1,024 − 50 = 974 spare, so a leak of 5 sockets a minute exhausts them in 974 / 5 ≈ 195 minutes, a little over three hours.

That interval is the reason the failure pattern is so recognisable: the service is healthy after a deploy, dies in the small hours, restarts clean, and dies again about three hours later. Restarting resets the count and nothing else, so the incident recurs on a fixed cycle that tracks the leak rate rather than traffic. Once accept() starts returning EMFILE the service is refusing connections while CPU, memory and every application metric look perfectly healthy.

The hunt:

ss -tanp state close-wait | awk '{print $6}' | sort | uniq -c | sort -rn
lsof -nP -p $PID | awk '/TCP/ {print $9}' | sed 's/.*->//' | sort | uniq -c

Group by process to find the offender, group by peer to find the closer. Then correlate the growth curve with events: spikes after quiet windows point at idle-timeout eviction by a load balancer; steady linear growth points at exception paths; steps after each deploy point at shutdown draining gone wrong. A tcpdump on the app port settles it: you will see the peer's FIN arrive and no FIN ever leave.

Fixes in order: close on EOF and in finally blocks; set client pool idle eviction below upstream idle timeout (pool 30 seconds against an ALB 60); enable pool health checks that discard dead entries; export a close-wait gauge and alert on slope. Rolling restarts clear the symptom and teach nothing.

My position: investigate sustained unexplained CLOSE_WAIT growth against expected half-close behavior and workload volume. I would not reverse that even under pressure to "tune the kernel", because no sysctl exists for forgetting to call close.

What interviewers probe next

"Does tcp_fin_timeout help?" No. It bounds FIN_WAIT_2 for orphaned local closes. Repeating that myth here ends the follow-up round quickly.

"Why do these pile up after every deploy?" In-flight and pooled connections from the old generation get closed by peers during drain, and code that ignores EOF keeps them forever.

"How would you reproduce it deliberately?" Point a client at netcat or a proxy that half-closes after N seconds, then watch the state table fill until the fix lands.

Common mistakes

Restarting the service and calling it resolved, with no owner assigned to the actual missing close.

Tuning tcp_fin_timeout or tw recycle knobs, proving kernel knowledge was applied where an application bug lives.

Monitoring only error rates; the leading indicator is a socket-state gauge, and by the time errors fire you are minutes from EMFILE.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.