DevOpsInterviews logo
← Containers & Kubernetes / 06
easyNewRed HatTCSAccenture

A pod is not running. Walk me through telling Pending, ImagePullBackOff and CrashLoopBackOff apart.

Three states, three completely different causes, and one command that tells you which you are looking at. The state name says whether the problem is the scheduler, the registry or your own process.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Read the pod state first, because it names the layer. Pending covers setup before containers are ready to run, including scheduling and image pulls. ImagePullBackOff identifies a failed image fetch; CrashLoopBackOff identifies delayed restarts after repeated exits. Then kubectl describe pod for the first two and kubectl logs --previous for the third.

How to approach it

Say that the state tells you which component is unhappy, then go to the evidence for that state specifically. The whole triage turns on reading the output rather than restarting and hoping, because the events already name the reason and the restart discards them. Naming --previous on the logs command is a small detail that signals real use.

A strong answer

The one command to start with, every time:

kubectl describe pod -n prod checkout-7d9f

The Events at the bottom are the answer to nearly all of these, and they are the part people scroll past.

Pending means the pod has been accepted but container setup is incomplete. First check whether spec.nodeName is set. For an unassigned pod, inspect scheduler events for causes such as these. The pod's resource requests exceed what any node has free (0/12 nodes are available: insufficient memory), which is a claim about what it asked for rather than what it would have used. A node selector, affinity rule or taint that nothing satisfies. A PersistentVolumeClaim that has not bound, so there is no volume to attach. Or the cluster genuinely has no nodes in a usable state. The fix follows from which one: add capacity, relax the constraint, or fix the storage.

ImagePullBackOff (and ErrImagePull just before it) means a node tried to pull the image and failed. The event carries the real error and the four causes look nothing alike: a typo in the image name or tag, a private registry with no valid imagePullSecret, a rate limit from a public registry, or a tag that was deleted. "BackOff" means the kubelet is now waiting between retries, and that wait grows to around five minutes, which is why a fixed credential can look like it did not work. After fixing the cause, observe the next retry. Replace a controller-managed pod only when the workload can tolerate it and you have preserved the evidence.

CrashLoopBackOff identifies repeated container exits and restart backoff. The container started and then exited, and Kubernetes restarted it, repeatedly, with a growing delay. The container ran, so the image and the scheduling were fine. Get the logs from the run that died:

kubectl logs -n prod checkout-7d9f --previous

Without --previous you get the logs of the current attempt, which may be a container that has not failed yet or has not written anything. The usual causes are a missing environment variable or config file, a dependency it cannot reach at startup, a failing migration, or a command that completes successfully and exits, since a container that exits 0 in a Deployment is still restarted.

Two states worth adding because they get confused with these. ContainerCreating that never progresses is usually a volume failing to mount or a Secret that does not exist. And a pod that is Running but never Ready is a failing readiness probe, which means it takes no traffic while looking healthy in a list.

kubectl get pods -n prod            # STATUS and READY columns
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Compare the phase with the container's waiting reason

These read-only commands distinguish pod placement from container startup:

kubectl -n prod get pod checkout-7d9f -o wide
kubectl -n prod get pod checkout-7d9f -o yaml
kubectl -n prod logs checkout-7d9f -c checkout --previous

In an example incident, status.phase is Pending, spec.nodeName is populated, and a container's waiting reason is ImagePullBackOff. Scheduling has already succeeded. Investigate that node's registry access and the exact pull error instead of adding cluster capacity.

In a second case, the container repeatedly exits with code 0 because the Deployment runs a command that performs one task and finishes. That suggests a workload/controller mismatch, not a memory failure. A third case can show a liveness probe killing a slow-starting process. Read the termination reason alongside events before assigning the problem to application code. Pod lifecycle distinguishes phases from the status text printed by kubectl.

What interviewers probe next

"Restart count is 200 but the pod is Running. What does that mean?" The count records past restarts. Compare its change over time and the last termination timestamp; a high historical count alone does not prove an active loop.

"Why does the backoff grow?" To stop a broken pod hammering a registry or a database. It doubles to a cap of about five minutes, which is also why fixing the underlying problem does not appear to help immediately.

"What if there are no logs at all?" The process died before writing anything, or it wrote to a file rather than stdout. Check the exit code in describe (137 is SIGKILL, often out of memory; 1 is an application error) and the termination message.

Common mistakes

Deleting the pod as a first move, which destroys the events and the previous container's logs, which were the evidence.

Reading kubectl logs without --previous on a crash loop and concluding there is nothing in the logs.

Treating ImagePullBackOff as one condition and trying a credential fix when the actual event says rate limit or manifest unknown.

Ignoring the Events section of describe, which usually states the cause in one sentence.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.