DevOpsInterviews logo
← Incident Response & Production Debugging / 03
medium★ EssentialGoogleUberFlipkart

A pod is in CrashLoopBackOff and the logs are empty. Debug it, step by step.

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Empty logs can mean an early failure, the wrong container instance, buffering or a different log destination. Read the last terminated state and events. Exit 137 commonly means SIGKILL, which can come from OOM handling or another actor; exit codes are clues, not unique diagnoses.

How to approach it

Narrate a hypothesis and the observation that would eliminate it, rather than listing every command you know. Assume the description is incomplete and ask for specific output rather than reasoning from the summary. Say what you would check first and why that first.

A strong answer

First, get the exit code and the events, because they partition the whole search space:

kubectl describe pod api-7f9c
kubectl get pod api-7f9c -o jsonpath='{.status.containerStatuses[?(@.name=="api")].lastState.terminated}'
kubectl logs api-7f9c -c api --previous   # the crashed instance, not the restarting one

kubectl logs without --previous reads the container that is currently backing off, which frequently has produced nothing yet. That alone explains a large share of "the logs are empty" reports.

Now branch on what comes back.

Exit code 137 with reason OOMKilled: the kernel killed it, but establish whether this was a cgroup OOM or a node-wide OOM. Check the node kernel log and cgroup memory events before changing the limit. A leak or a changed working set may be the cause. A JVM or Node process that reads host memory rather than its cgroup limit will happily size a heap larger than the container is allowed, and dies during warm-up every time.

Exit code 1 or 2 with no output: check initialization and logging configuration; the exit code alone does not establish either cause. A missing environment variable, a Secret or ConfigMap that does not exist in this namespace, or a dependency it connects to at startup and cannot reach. kubectl describe events will show FailedMount if it is a volume, and silence if it is a network dependency.

Exit code 126 or 127: the entrypoint is not executable or does not exist. Wrong architecture image on an arm node, a missing shared library in a distroless base, a script without the executable bit.

Repeated Killing events can identify a probe-triggered restart; inspect their messages and the terminated state together. The container starts, the probe fires before the app is ready, kubelet kills it, and the cycle repeats forever. The tell is that the restart interval matches the probe's period and threshold rather than the app's startup time. The fix is a startupProbe with a generous failure threshold, so slow starts are tolerated while liveness stays aggressive once running.

If none of the above is conclusive, use a debug container, or start a disposable copy if the application image actually contains a shell:

kubectl debug -it api-7f9c --image=busybox --target=api
kubectl run tmp --rm -it --image=myapp:1.4.2 --command -- sh

The BusyBox shell belongs to the debug image. Sharing a process namespace does not give it the target container's root filesystem; inspect target paths through /proc only where runtime support and permissions allow. A distroless application image may have no sh at all.

Then change one thing supported by the evidence. In production the order is mitigate first: roll back to the last known-good image tag, get the service healthy, and diagnose the bad image outside the critical path.

Reading a crash loop when the logs come back empty GET THE EVIDENCE TEST THE HYPOTHESIS 1 Ask for the previous logs not the container backing off 2 Take the exit code from the last terminated state 3 Read the events correlate timing and messages 4 137 with OOMKilled the limit, or the node 5 137 without it check who sent SIGKILL 6 1 or 2 the application chose to exit 7 126 or 127 entrypoint, not the program 8 Nothing at all check instance and destination Select the container by name and record its pod UID. Previous logs belong to a retained container attempt in that pod, not a replacement pod. Empty output can mean an early exit, buffering, a different destination or no retained attempt. Correlate runtime and pod events before choosing a recovery action.

Preserve the identity of the failed attempt

In an example rollout, api and proxy share a pod. Reading container index zero happens to inspect the proxy, while the application is the one restarting. Select the container by name and capture the pod UID, image digest, restart count and last termination timestamp. A replacement pod with a similar name does not retain the old pod's previous logs.

Suppose the application exited at 10:03:20 with code 137. An event at the same time reports a liveness failure, but that correlation alone is insufficient: check whether kubelet initiated termination and whether the process exceeded its shutdown grace period. Separately check OOM evidence. Both paths can end with SIGKILL and need different changes.

Before a recovery rollout, save the useful events and status in the incident record without copying Secret values. After recovery, compare restart counts over a representative startup interval and exercise the application request path. One successful start is weak evidence if the failure only occurs during cache warm-up.

What interviewers probe next

"Why did raising the memory limit fix it in staging and not production?" Because production has a different working set, or because the limit change moved the pod to a node where it now cannot be scheduled, or because the real cause is a leak and staging never runs long enough to reach it.

"How do you tell an OOMKill of your container from a node under memory pressure?" A cgroup OOM and a node-wide kernel OOM can both produce OOMKilled. Kubelet node-pressure eviction is a separate path with eviction events. Correlate termination details, cgroup memory events and node kernel logs.

"What would you check if kubectl logs --previous says the container has not restarted?" Confirm the container name, pod UID and restart count. There may be no retained previous instance, or you may be looking at a replacement pod; use events and the runtime's available logs.

Common mistakes

Reading logs of the running container and concluding there is nothing to see. --previous is the entire trick.

Jumping straight to raising the memory limit because 137 was mentioned once in a runbook. Confirm OOMKilled first; 137 is also what you get from an external SIGKILL.

Removing the liveness probe as a permanent fix rather than as a diagnostic. Removing it to prove the probe is the cause is fine. Leaving it removed is how a hung process runs for a week.

Listing fifteen commands with no hypothesis. Explain what each command can establish before choosing the next one.

References

Kubernetes pod lifecycle.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.