The control loop: the one idea Kubernetes is built on
Kubernetes controllers reconcile desired and observed state through the API. That model explains many stuck objects, while runtime operations still use direct calls to container runtimes, networking and storage systems.
TL;DR: Kubernetes is a set of independent controllers, each watching the API server and reconciling one difference between desired and actual state. Independent reconciliation lets one controller resume after another recovers. Failures can appear as stalled progress, status conditions, events or component errors.
The shape
A controller does exactly this, forever:
watch(resource) # observe desired state
observe(reality) # observe actual state
if desired != actual:
act to close the gap
The deployment controller watches Deployments and creates ReplicaSets. The replicaset controller watches ReplicaSets and creates or deletes Pods. The scheduler watches Pods with no nodeName and writes one. The kubelet watches Pods bound to its node and makes containers exist. The EndpointSlice controller watches Pods and Services and updates EndpointSlices.
These controllers coordinate desired state through API objects. Their actions also use other interfaces: kubelet calls the container runtime, and storage controllers call storage APIs. Etcd stores Kubernetes API state; application data, container images and node-local state live elsewhere.
Why the architecture was chosen
Three consequences fall out of it, and all three are the actual reason the model is used.
Level-triggered rather than edge-triggered. Events usually trigger reconciliation, which checks the current difference rather than blindly replaying a transition. If a controller is restarted, or misses an event entirely, it recomputes the difference on its next pass and still converges. That is why Kubernetes tolerates its own components being down.
The control plane is not on the data path. Kill the scheduler and running pods keep serving traffic, because the kubelet does not need the scheduler for anything already bound. New pods stay Pending. During an API-server outage, many existing workloads and programmed network paths keep working. Workloads that need live API reads, fresh credentials or new storage/network setup may fail; continued service is conditional on their dependencies.
Extensibility uses the same pattern. A custom resource plus a controller is architecturally identical to a built-in one. That is what an operator is: a control loop that encodes the operational knowledge for a specific piece of software, using the same API conventions. It still needs carefully scoped RBAC, retry handling and operational support.
What this predicts about failures
A broken reconciliation loop often leaves an object waiting for progress. Look for status conditions and events explaining that wait, then inspect the responsible component's logs or external dependency.
A Pending pod may be unscheduled or already assigned to a node while images, volumes or init containers are prepared. Check PodScheduled and spec.nodeName first. Terminating is a display status for deletion in progress, not a Pod phase: grace periods, unreachable nodes and finalizers can all delay deletion. A PVC may be protected while a pod still uses it; inspect references and finalizers before removing any protection. A Deployment reporting fewer ready replicas than desired means some loop downstream is blocked and the status is telling you where.
The diagnostic instinct that follows: when something is stuck, find the controller responsible for that transition and read its view of the world, which is almost always in the object's events and status conditions rather than in a log file.
Worked example: three desired replicas, only two serving
A Deployment declares three replicas. Its ReplicaSet has created three Pod objects, but one pod has no assigned node. The Deployment controller has not necessarily failed: the downstream scheduling step is blocked. Inspect that pod's scheduling condition and events to determine whether requests, affinity or another constraint prevents placement.
After capacity is added, the scheduler assigns a node. The pod can still wait while an image is pulled or a volume is mounted. Once the container starts, readiness may remain false because the application cannot connect to its dependency. These are separate observations along the chain. Restarting every controller would discard evidence without changing the missing capacity or unavailable dependency.
For each transition, name the owner, its input and the evidence of progress:
| Observation | Next evidence to inspect |
|---|---|
| ReplicaSet cannot create a pod | ReplicaSet events and admission rejection |
| Pod exists without a node | Scheduling conditions and feasible node capacity |
| Assigned pod has no running container | Kubelet events, image and volume preparation |
| Running container is not Ready | Probe results and application dependency health |
Repeated reconciliation can recover once the blocking condition changes. It cannot manufacture capacity or decide that an unsafe storage handoff is acceptable.
Self-check
Three situations, each testing the same model from a different angle. First, the scheduler crashes and is down for ten minutes; during that time twelve new Pods are created. What state is the cluster in when it comes back, and what did it need to have persisted in order to recover? Second, a Deployment says 3 replicas and reports 2 ready, and a colleague proposes restarting the deployment controller. Third, a PersistentVolumeClaim has been Terminating for an hour and someone suggests removing its finalizer to clear it. Work through all three before reading on.
The scheduler outage is the case the architecture is designed for, and the answer turns on what a Pod object is. The twelve new Pods were written to the API by their ReplicaSet controllers and sit there with no nodeName, which is not a queued request that can be lost; it is the desired state, recorded durably. When the scheduler returns it lists Pods needing placement and evaluates them against the constraints that exist now, which may differ from those at creation time. So it needed no durable copy of the ten minutes of events it missed, and this is precisely what level-triggered means: the difference is recomputed rather than replayed. What recovery does still depend on is ordinary and worth naming, being API availability, the scheduler's own permissions, and a feasible node actually existing. A Pod that is unschedulable because the cluster is full stays Pending, and the scheduler coming back changes nothing about that.
Restarting the deployment controller is the wrong move, and the reason generalises to most stuck objects. The Deployment controller's job is to produce a ReplicaSet with the right number of Pod objects, and by reporting 2 of 3 ready it is telling you it already did that and is waiting on something downstream. The blockage is in whichever transition has not completed: the Pod may have no node, in which case the scheduler and node capacity are the subject; it may have a node and no running container, which points at image pull or volume mounting; or it may be running and failing readiness, which is the application or its dependency. Each of those has a different owner. Restarting controllers discards the events and conditions that say which one it is, and reconciliation will resume without having changed the missing capacity or the unavailable dependency.
The finalizer is the most dangerous of the three, because removing it appears to work. A finalizer is a claim that some cleanup must happen before the object is allowed to disappear, and deleting the marker does not perform the cleanup it represented; it declares the cleanup unnecessary. The likely causes are a Pod still referencing the PVC, which is protection working as designed, or a storage controller that cannot complete detachment. So the order is to find what still references it and whether its controller is reporting an error, and to treat finalizer removal as an operation that may leave a volume attached to a node or orphan storage you continue to pay for.
The instinct all three share: when something is stuck, identify the controller responsible for the transition that has not happened, and read its view of the world, which is almost always in the object's events and status conditions rather than in a log file.
Operators and CRDs applies this loop to a custom API, including stale status generations, external cleanup and storage-version changes.
Next: GitOps reconciliation is this pattern applied one level up to deployments, and requests are what the scheduler believes covers why a feasible placement may not exist.