DevOpsInterviews logo
← Incident Response & Production Debugging / 05
medium★ EssentialUberShopifyFlipkart

A service returns 503 but every pod behind it looks healthy. Debug it.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Identify which HTTP component emitted the 503, then inspect every EndpointSlice for the Service, including readiness conditions and ports. A Kubernetes Service does not itself generate HTTP responses. Missing usable backends, a wrong target port or an application/proxy failure can all produce this symptom.

How to approach it

Name the one command that halves the problem in the first sentence, then branch. This is a question about how traffic actually reaches a pod, and every 503 is one specific link in that chain being empty or broken.

A strong answer

A typical route goes through an ingress proxy to a backend pod, using Service and EndpointSlice data for discovery. Some controllers route directly to pod IPs. Use response headers and proxy/application logs to locate the 503 emitter, then inspect the backend set:

kubectl get endpointslices -l kubernetes.io/service-name=api -o yaml
kubectl get pods -l app=api -o wide

Inspect all slices in the Service's namespace. Distinguish no matching endpoints, endpoints marked unready, and ready endpoints advertising the wrong port.

Readiness is failing. Pods can be Running and not Ready indefinitely. EndpointSlices can still contain their addresses with conditions.ready: false; consumers normally exclude them from regular traffic. publishNotReadyAddresses and terminating-endpoint handling are exceptions to check. kubectl get pods shows 1/1 versus 0/1 in the READY column, which people skim past because the STATUS column says Running. A readiness probe hitting a path that returns 404, or a timeout shorter than the dependency it checks, does this.

The selector matches nothing. The Service selects app=api and the deployment labels pods app=api-server. Nothing errors, because a selector matching zero pods is legal. Compare the two directly rather than trusting either.

The port mapping is wrong. targetPort must match the port the container actually listens on, and a named port must exist in the pod spec. A numeric targetPort: 8080 can remain advertised in a populated EndpointSlice even when the application listens on 3000. That causes connection failures; the proxy determines whether the user sees 503, another HTTP error or a timeout.

If endpoints are ready and ports are correct, test reachability from the actual caller and inspect the proxy. The Ingress controller may have a stale or misconfigured backend, the ingress class may be wrong so no controller owns the rule, or a NetworkPolicy may be dropping traffic from the ingress namespace to the pods. That last one is a favourite because the pod is healthy and unreachable.

The version that appears during a rollout is worth naming unprompted: readiness and termination conditions propagate asynchronously, so a terminating pod can receive traffic while consumers update routing. Coordinate readiness, graceful shutdown and connection draining; a measured preStop delay may cover propagation, but a fixed sleep alone does not guarantee safe draining.

Trace a 503 from the emitting proxy to its backend DISTINGUISH THE EVIDENCE 1 Who emitted the 503 headers, and proxy logs 2 Read the EndpointSlices all of them, in that namespace 3 No endpoints at all inspect selectors and slice ownership 4 Endpoints, marked unready readiness is failing 5 Ready, but wrong port compare targetPort with listener 6 Check the READY column 0/1, beside Running 7 Read the probe path, and its timeout 8 Test the identified boundary then verify the user request A pod can be Running and not Ready indefinitely. The slice can still carry its address with the ready condition false, and consumers normally exclude it from ordinary traffic. The column people skim past, because the status column says Running and the eye stops there. A readiness probe hitting a path that returns 404, or a timeout shorter than the dependency it checks, produces exactly this.

Keep the request identical across diagnostic hops

Suppose the failing request is POST /checkout with a tenant header, while a pod port-forward test uses GET /healthz. The successful health response establishes little about checkout. Repeat an authorized, side-effect-safe equivalent of the failing request with the same host, path and relevant routing headers, first through the public path and then from the ingress caller's network context.

If only one endpoint fails, compare its effective configuration and listening port with the other replicas. A ready EndpointSlice entry is discovery state, not a live transport test. If every endpoint works directly but the proxy fails, inspect its current upstream configuration, connection errors and policy decisions.

Capture endpoint and route state before restarting a controller. After mitigation, verify both successful requests and the expected denial of unauthorized requests. Broadening a NetworkPolicy until checkout works could accidentally make an internal administrative endpoint reachable too.

What interviewers probe next

"How would you tell a Service problem from an application problem?" Port-forward straight to a pod and bypass the Service. A successful response checks that pod and port, but port-forward bypasses parts of the network path and does not prove every replica or normal request succeeds. Compare the same request from the ingress network path.

"Everything is Ready and it still 503s intermittently." Look at rollout timing, connection draining, and whether the readiness probe reflects real dependency health rather than just process liveness.

"Why not just restart the deployment?" It often clears the symptom and destroys the evidence. Fine as mitigation once you have captured the endpoint state, not before.

Common mistakes

Skipping the endpoints check, which is the single fastest way to halve the search space.

Reading STATUS and ignoring READY. The distinction is the entire question.

Forgetting NetworkPolicy, which produces a healthy pod that cannot be reached and no error anywhere obvious.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.