DevOpsInterviews logo
← Containers & Kubernetes / 16
medium★ EssentialUberShopifyRed Hat

Liveness, readiness and startup probes: what does each one do when it fails, and how do you get them wrong?

Everyone can recite the three. Explain what each failure causes, and why a badly configured liveness probe is more dangerous than none at all.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Readiness failure marks the pod unready for ordinary Service routing and keeps it running. Liveness failure kills and restarts the container. Startup failure kills it too, but it suspends the other two while the app boots. The dangerous one is liveness, because a probe that fails under load restarts healthy pods and turns a slowdown into an outage.

How to approach it

Lead with the consequence of each failure rather than the definition, because the consequences are what make them different. Then make the argument that liveness should be conservative, which is the opinionated part.

A strong answer

Readiness answers "should I get traffic right now?" On failure the matching endpoint is marked unready. Ordinary Service routing stops selecting it for new traffic once the update propagates, but existing connections and direct pod-IP requests can continue. This is the probe that should reflect real ability to serve, including a dependency you cannot work without. Avoid aggressive checks against shared dependencies: one database outage can mark every replica unready and remove all service capacity. Check whether removing the pod actually improves the request outcome.

Liveness answers "is this process wedged beyond recovery?" On failure the kubelet kills the container and the restart policy applies. This is the dangerous one. If your liveness probe hits an endpoint that also does real work, then under load it times out, the kubelet kills a pod that was merely busy, load shifts to the remaining pods, they get busier, and you have built a cascading restart loop that takes the service down. A liveness probe should test almost nothing: that the process is alive and its event loop is turning. If you cannot name a failure mode where the process is running but permanently unrecoverable, you do not need a liveness probe at all, and no probe is safer than a bad one.

Startup exists because those two have incompatible requirements during boot. A JVM that takes 90 seconds to warm up needs a long liveness timeout, but a long timeout means a genuinely wedged process is not restarted for 90 seconds forever after. The startup probe resolves it: while it is failing, liveness and readiness are suspended, and once it passes it stops until the container is restarted. So you get a generous boot allowance and an aggressive steady-state check. Size that allowance by multiplying rather than guessing at it. The startup budget is periodSeconds × failureThreshold, so a probe every 5 seconds with a threshold of 30 grants 5 × 30 = 150 seconds before the container is killed, plus any initialDelaySeconds. For a service that warms up in about 90 seconds that is the right shape: roughly 60 percent margin over the observed time, because the boot is slower on a cold node, during an image pull, or when a dependency is briefly slow.

Sizing it tight is what produces the failure this probe exists to prevent. An allowance of exactly 90 seconds means a boot that takes 95 gets killed, the restart takes another 90-plus, and the pod enters a crash loop that reads as an application defect while the application is merely being interrupted. Meanwhile the steady-state liveness probe can stay aggressive, say periodSeconds: 10 with failureThreshold: 3 for 30-second detection, and the two numbers no longer trade against each other, which is the entire point of having a third probe. Kubernetes probe configuration.

The configuration that most often causes an incident: liveness and readiness pointed at the same handler. That handler checks the database, the database gets slow, every pod fails liveness simultaneously, and the entire deployment restarts at once. Choose probe semantics from the action they trigger. Sharing an endpoint is inappropriate when it makes restart depend on an external failure that restart cannot repair.

The three probes, in the order they actually apply BOOT STEADY STATE 1 Container starts begin this container lifecycle 2 Startup probe runs liveness and readiness suspended 3 It passes, once until this container restarts 4 Readiness takes over should I get traffic now 5 Liveness takes over is this process wedged 6 Readiness fails the endpoint stops being picked 7 Liveness fails the container is killed 8 Restart policy applies and the loop can begin Startup allows a separate boot budget. After success, steady-state checks apply. A restarted container begins probing again. Avoid aggressive checks against a shared dependency here: one database outage can mark every replica unready and remove all capacity. Ask whether removing this pod actually improves the request's outcome. A dependency-sensitive liveness handler can restart every replica during a shared outage. Test whether restart repairs the failure before using it as the probe action.

Probe exercise: a slow dependency triggers a restart storm

A fictional API has ten replicas and a liveness handler that queries its shared database. Database latency exceeds the probe timeout. Restarting each API instance closes pools and causes new connection bursts, worsening the original bottleneck. Replace that liveness dependency with a check tied to a failure restart can repair, and use application metrics to detect dependency degradation.

For readiness, decide whether removing a replica improves outcomes. If all replicas share the same unavailable dependency, removing every endpoint may only change the failure presented to clients. A degraded response or bounded rejection path can be preferable, depending on the service contract.

Rehearse boot and steady-state failures separately. Startup success disables that startup probe for the current container execution; a container restart begins its lifecycle again. Record observed probe timings instead of assuming threshold multiplied by period is an exact restart deadline. Verify new-connection routing after readiness changes, while accounting for existing connections and propagation.

What interviewers probe next

"When would you use no liveness probe?" When there is no identified running failure that a restart can repair. If a crash exits the process, the restart policy already handles it. Liveness is for the deadlock case specifically.

"How do the probes interact with a rolling update?" Readiness gates it. A new pod that never becomes Ready blocks the rollout, which is a feature: it stops a broken version replacing a working one.

"What about exec probes?" They fork a process on every check. At high frequency across many pods that is real overhead, and a slow exec probe can pile up. Prefer HTTP or gRPC where you have the choice.

Common mistakes

Using a dependency-sensitive readiness handler for liveness without checking whether restart helps.

Making liveness check dependencies. If the database is down, restarting your pod does not help and makes recovery slower.

Not knowing the startup probe exists, then using a long initialDelaySeconds without a startup probe. Initial delay applies only after container startup; it does not delay every steady-state failure detection.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.