DevOpsInterviews logo
← Containers & Kubernetes / 11
medium★ EssentialGoogleMetaRed Hat

Walk me through everything that happens between kubectl apply and a running pod.

The single most common Kubernetes interview question, and the one where a vague answer is most obvious. Every stage you name is a place the interviewer can ask you to debug.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: The API server authenticates, admits and persists the object to etcd, and that is the whole of the write path. Everything after it is controllers watching for a difference and acting: the deployment and replicaset controllers create the pod, the scheduler binds it to a node, the kubelet pulls and starts it, and the CNI and kube-proxy make it reachable.

How to approach it

Name the stages in order and keep each to a sentence. The value of the answer is that it gives you a fault tree: for every stage you name, you know what "stuck here" looks like. Finish by mapping each stalled transition to the component you would inspect.

A strong answer

The request hits the API server, which authenticates you, authorises the verb against RBAC, then runs admission. Mutating admission webhooks and defaulting run first, so this is where a sidecar gets injected or a default resource limit gets stamped on. Validating admission and policy engines such as Kyverno or a Pod Security admission run next and can reject outright. If it survives, the object is persisted to etcd and the API server returns success. Your kubectl apply is now done, and nothing is running.

Everything after this is a control loop. The deployment controller sees a Deployment with no matching ReplicaSet and creates one. The replicaset controller sees a ReplicaSet whose desired replica count exceeds observed and creates Pod objects, which land in etcd with nodeName empty.

The scheduler watches for unbound pods. It filters nodes that cannot run the pod at all (insufficient allocatable resources, unsatisfied node selector or affinity, an untolerated taint, an unbindable volume) and then scores the survivors and writes a binding. If nothing survives filtering, the pod sits in Pending and the events explain which predicate failed.

The kubelet on the bound node takes over. Through CRI it asks the container runtime to create the pod sandbox; the runtime normally creates the network namespace and invokes CNI to configure networking and allocate an IP. Storage controllers and the kubelet coordinate CSI attachment and mounting. The runtime pulls images and starts containers. Ordinary init containers complete in sequence; restartable native sidecars can continue running after their startup condition allows the next container. EndpointSlices can contain an unready pod already: readiness updates endpoint conditions, which service proxies normally use to select traffic destinations. Pod lifecycle, EndpointSlice conditions.

What happens between kubectl apply and a running pod API SERVER: THE WHOLE WRITE PATH CONTROLLERS WATCH AND ACT ON THE NODE reconcile: drift starts this again 1 kubectl apply POST the desired object 2 Authenticate who is calling 3 Authorise RBAC: may they do this verb 4 Mutating admission webhooks and defaulting 5 Validating admission policy can reject outright 6 Persist to etcd apply returns success 7 Deployment controller creates a ReplicaSet 8 ReplicaSet controller creates Pod objects 9 Scheduler filter, score, write a binding 10 kubelet on that node sandbox, CNI, CSI, image pull 11 Containers start init, then app containers 12 Readiness passes EndpointSlice updated Where a sidecar appears that nobody wrote in the manifest, and where a default resource limit is stamped on. Your command is finished here, and nothing is running. Every stage below is a controller noticing a difference, not a component calling the next one. Filters out nodes that cannot run it at all, scores what survives, and writes a node name. It starts nothing. Pending: nothing survived filtering The pod IP is allocated here, by CNI, on the node. Not by the API server and not by the scheduler. ContainerCreating: CNI, CSI or pull CrashLoopBackOff: it started and died, so read the last terminated state A pod can be in the EndpointSlice before it is ready. Readiness is what makes the proxy select it. Running, no traffic: readiness or selector

The debugging payoff is to find the transition that has not completed. Pending covers both unscheduled pods and scheduled pods whose containers are not yet set up. Inspect the PodScheduled condition and events before blaming the scheduler. ContainerCreating points toward setup such as networking, volumes or image retrieval. CrashLoopBackOff is a container restart backoff, so inspect the last terminated state and previous logs. A Running pod can still fail readiness or have a broken traffic path.

Use this illustrative case to turn the component list into a diagnosis. The Deployment is accepted, a new ReplicaSet exists, but its new pod is Pending. The pod's PodScheduled condition is false, and its event reports insufficient memory. The API write and controller work have happened; restarting the application cannot repair a scheduling constraint because the new container has not started. Compare its memory request with allocatable, already-requested capacity on eligible nodes. Aggregate free memory across the cluster does not prove that one eligible node can fit it.

EvidenceNext checkWhy
No new ReplicaSetDeployment conditions and controller eventsReconciliation or admission may have failed
PodScheduled falsePod events, requests and node constraintsThe scheduler has not found a usable placement
Scheduled, container waitingWaiting reason, image and volume eventsPlacement succeeded; setup has not
Container repeatedly terminatesExit reason and previous container logsStartup or runtime execution is failing
Running but no successful requestsReadiness, Service selector, endpoints and network pathProcess state does not establish service reachability

For the last row, check whether the Service selects this pod and whether the endpoint is ready before investigating routing or network policy. A service proxy may be kube-proxy or another implementation; naming kube-proxy should not imply that every cluster uses it. Explain what evidence moves you to the next boundary rather than reciting every component at once.

What interviewers probe next

"Where is the pod's IP assigned?" By the CNI plugin, on the node, into the pod's network namespace. Not by the API server, and not by the scheduler.

"What if the scheduler is down?" Existing pods keep running because the kubelet does not need it. New pods stay Pending indefinitely. This is a good illustration of why the control plane is not on the data path.

"Where would an injected sidecar come from?" A mutating admission webhook, which is why it appears in the object even though nobody wrote it in the manifest.

Common mistakes

Describing it as a linear pipeline where each component calls the next. Controllers coordinate asynchronously through API state, while kubelet, runtimes and plugins also make direct API calls. Controllers watch the API server and reconcile, which is why the failure modes are "stuck" rather than "errored".

Skipping admission. It is where a surprising share of real production behaviour is introduced, which is why a change review that skips it is reviewing the wrong artifact.

Saying the scheduler "starts the pod". The scheduler only writes a node name. The kubelet starts things.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.