TL;DR: A 504 comes from a gateway that gave up waiting, so it is emitted by a component your backend metrics do not describe. The request either never reached the backend or reached it and the response did not return in time. Look at the hops between: queueing at the ingress, connection pool exhaustion, NAT port capacity, DNS, and the difference between server-side latency and time in queue.
How to approach it
Point out immediately that backend metrics measure requests the backend accepted, so a request that never got accepted is invisible to them. That reframing is the answer; everything after is where to look.
A strong answer
Your service dashboard reports duration for requests it handled. If the request sat in an accept queue for nine seconds and then completed in 40 milliseconds, your dashboard shows 40 milliseconds and the user saw a timeout. That systematic blindness is why this question is hard and why "the backend is fine" is usually a true and misleading statement.
Work the hops in order.
The gateway's own view. The proxy or load balancer records both upstream response time and total time. Check the proxy's exact field definitions: high total and low upstream time can indicate client upload, downstream transmission, queueing or other overhead. Connection setup may already be included in upstream timing. This one comparison resolves a large share of these incidents.
Accept queue and worker saturation. If every worker or thread is busy, new connections queue in the kernel and wait. The application never sees them, so latency looks fine, and netstat -s overflow counters or the runtime's own queue depth is the evidence.
A simplified fluid-queue example shows how a sustained deficit accumulates. Assume constant service time, enough queue capacity and no abandonment: Fifty workers each handling a request in 40 milliseconds gives a capacity of 50 / 0.04 = 1,250 requests a second. Arrivals at 1,400 exceed that by 1,400 − 1,250 = 150 a second, so after a minute of it roughly 150 × 60 = 9,000 requests are queued in this model, and a request joining the back of that queue waits 9,000 / 1,250 = 7.2 seconds before a worker ever looks at it. Then it completes in 40 milliseconds, and that 40 milliseconds is the only number your dashboard records.
Notice what the arithmetic says about the fix. The service is 12 percent short of capacity and produces multi-second waits, because the queue integrates the shortfall over time rather than reflecting it instantaneously. A small, sustained deficit is enough, which is why this arrives without any obvious traffic spike. Compare measured queue growth with demand before applying this model to the incident.
Connection pool exhaustion. Between the gateway and the service, or between the service and the database. A pool sized for normal load runs out under a spike and callers block waiting for a connection. Time is spent waiting for a pool slot, which almost nobody instruments, so it appears as unexplained latency at the caller and nothing at all downstream.
NAT and conntrack. In cloud environments a NAT gateway has a port allocation per destination and a conntrack table with a size limit. Exhaust either and new connections are dropped silently, which presents as intermittent timeouts under load with every service healthy. This is a favourite because nothing in the application layer indicates it.
DNS. Resolution happens before connection establishment and may be excluded from backend timings; client-side DNS instrumentation can measure it. An intermittent five-second stall can be a resolver retry timeout, and it produces exactly this symptom if the client timeout is above it.
Timeout mismatch. If the gateway waits 10 seconds and the backend's own timeout to the database is 30, the gateway gives up first and the backend keeps working on a request nobody is waiting for, consuming a worker. Timeouts should decrease as you go inward, and getting this backwards converts a slow dependency into a cascading failure.
The measurement that settles it: instrument client-perceived latency, either from the edge or from a synthetic probe running the full path. Until you measure the whole path, you are comparing one segment against a complaint about the journey.
Build a deadline budget for the failed request
For an illustrative request with a two-second caller deadline, measure 300 ms waiting for a pool slot, 100 ms establishing the connection and 1.4 seconds in the dependency. Only 200 ms remains for response transfer and other overhead. Retrying the dependency with a fresh two-second timeout can exceed the original caller's budget even if each individual timer appears reasonable.
Propagate the remaining deadline where the stack supports it, and verify that cancellation releases workers and checked-out connections. A gateway returning 504 does not necessarily cancel the backend's work. Count requests still executing after their callers have left; this explains why increasing a timeout can worsen saturation.
After mitigation, check success rate together with total duration and queue wait for the same request population. A drop in 504s accompanied by longer queues may mean the system merely waits longer before failing. Recovery should restore completion within the intended deadline under representative demand.
What interviewers probe next
"Difference between 502, 503 and 504?" HTTP defines 502 as an invalid upstream response, 503 as temporary inability to serve, and 504 as failure to receive a timely upstream response. Proxies map concrete failures differently: a 504 can include connection establishment timeout, and a 503 does not uniquely mean no healthy backends. Read the emitting proxy's reason code.
"How would you test the connection pool theory?" Measure acquisition wait, active connections and backend saturation first. A bounded canary pool increase can test the hypothesis if the backend has headroom, but can also move or worsen the bottleneck.
"Why intermittent rather than constant?" Almost every cause here is a capacity limit, and limits are only reached at peak.
Common mistakes
Accepting "the backend is fine" as evidence. It measures accepted requests only.
Not comparing upstream time to total time at the proxy, which is the fastest available discriminator.
Raising timeouts to make the error disappear, which converts fast failures into slow ones and hides the cause.
References