TL;DR: Every tracked flow occupies a conntrack entry until its protocol timeout expires, and the default TCP established timeout is five days. When the table fills, NEW packets are dropped while established flows continue, so the symptom is intermittent and load-dependent. Inspect table occupancy and failure counters, reduce unnecessary connection churn, then choose a capacity or timeout change based on the traffic.
How to approach it
Explain the asymmetry up front: packets for existing flows reuse their entries, while a new tracked flow needs admission, which is why existing sessions survive and new ones fail probabilistically. Then ask what this box does, because NAT or a NodePort service is what drags a host into tracking flows it does not care about.
A strong answer
Netfilter's connection tracker records state for flows that traverse its tracking hooks; explicitly untracked traffic and some eBPF paths use different handling. Routing alone does not require it, but SNAT, MASQUERADE, Kubernetes services and most load balancers do, because reply packets must find their way back. Each entry lives until its timeout: an established TCP entry defaults to 432000 seconds on many kernels, five days, if the entry remains established and receives no traffic; FIN/RST processing normally moves closed connections to shorter-lived states.
When net.netfilter.nf_conntrack_count reaches nf_conntrack_max, the kernel drops packets that would create entries. Established connections have no first packet any more, so they sail through. Health checks pass, dashboards stay green on success-rate graphs sampled from long-lived clients, and yet a fraction of new connections fail. That is precisely the intermittent signature in front of you.
Confirm before changing anything:
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
conntrack -S # per-CPU stats; insert_failed climbing means drops now
dmesg | grep nf_conntrack
The fixes are ordered by how much they cost.
Reduce connection churn. Check whether clients reuse connections and whether health checks or retries create unnecessary flows. Consider narrowly scoped NOTRACK rules only after verifying that the traffic needs neither NAT nor stateful firewall handling in either direction. Bypassing tracking on a Kubernetes node without that check can break service routing or policy enforcement.
The arithmetic that tells you which of these you need is occupancy, not rate. The table holds an entry for as long as its timeout, so steady-state entries are new flows per second multiplied by mean entry lifetime, using the steady-state relationship between arrival rate and mean residence time.
Put numbers on it. A node opening 2,000 new connections a second whose entries sit in TIME_WAIT for the common 120-second default holds about 2,000 × 120 = 240,000 entries in steady state. Against a nf_conntrack_max of 262,144 that is 240,000 / 262,144 ≈ 92 percent occupancy on an ordinary day, which is why the failure looks intermittent and arrives with a modest traffic bump rather than a surge.
The two levers then separate cleanly. Halving churn to 1,000 connections a second halves occupancy to 120,000. Cutting nf_conntrack_tcp_timeout_time_wait to 30 seconds instead gives 2,000 × 30 = 60,000, a fourfold reduction, which is why timeout tuning often outperforms raising the ceiling. Raising nf_conntrack_max to one million moves occupancy to 24 percent and changes nothing about the flow factory, but leaves the entry creation and retention behavior unchanged.
Tune timeouts from observed connection lifetimes. nf_conntrack_tcp_timeout_time_wait commonly defaults to 120s. Measure which states occupy the table before changing their timeouts, test retransmissions and idle sessions, and retain a rollback. Reducing established or closing-state timeouts to a few seconds can disrupt valid connections or lose NAT mappings.
Raise the limit, with the memory arithmetic. Memory per entry and hash-table behavior depend on kernel configuration and extensions. Measure slab memory and lookup cost on the deployed kernel, then budget headroom before increasing the limit. Follow the platform's supported hash-table configuration procedure.
On Kubernetes nodes, inspect service translation, SNAT and health-check traffic together. An eBPF dataplane may replace netfilter conntrack for some paths, but its own connection-tracking and NAT maps still have capacity limits; check the deployed dataplane and configuration.
Estimate occupancy from the observed state mix
The 240,000-entry example assumes all 2,000 new flows per second remain for an average of 120 seconds. Real tables contain different protocols and TCP states. Sample a suitably filtered listing and compare state counts with traffic rate before applying that estimate to capacity planning.
For example, reducing a TIME_WAIT timeout has little effect if most entries are long-lived established connections. Conversely, a burst of short-lived clients can fill closing states even when current application concurrency is low. Check both rate and retention. Keep diagnostic dumps bounded on an already stressed node.
After a reviewed change, verify new connections succeed, occupancy settles and failure-counter deltas stop growing. Also test valid idle sessions through the NAT path for longer than the proposed timeout. A lower count achieved by prematurely expiring legitimate mappings can trade connection-creation failures for broken existing sessions.
What interviewers probe next
"How do you find who is filling the table?" Use conntrack -S for read-only counters, then inspect a filtered conntrack -L listing by source or destination if the node can tolerate the dump. conntrack -D deletes matching entries; it is not a diagnostic counter command. Compare traffic growth and connection reuse before blaming a client.
"Why did monitoring miss it?" Because nothing errored where monitors look. Add insert_failed from conntrack -S as an alerting signal; correlate it with table occupancy, drop counters and connection failures rather than assuming it always gives advance warning.
"Is there a number where the table should sit?" Choose headroom from peak flow rate, retention and recovery time. No universal occupancy percentage predicts an outage; alert on failed admission and connection failures as well as capacity.
Common mistakes
Doubling nf_conntrack_max on every node in the fleet and moving on. Memory cost scales across every box, and the underlying flow factory keeps growing.
Assuming the firewall team's rules are unrelated. Any rule path that hits NAT pulls flows into the table even when the policy ends in accept.
Confusing this with SYN flood behaviour. A flood exhausts half-open queues and shows SYN drops; conntrack exhaustion drops NEW packets of all protocols including UDP, which is a useful differential when DNS starts failing alongside HTTP.