The cloud control plane is a dependency you forgot to design around

TL;DR: Separate what runs your traffic from what changes your infrastructure. A control-plane impairment can leave existing data-plane operations healthy. A regional outage can also damage both. Recovery that requires launching, scaling or reconfiguring must account for unavailable APIs. Pre-provision the capacity you would need, and prefer failovers that require no API call at all.

The symptom this explains

A region has a bad morning. Your instances are running, your load balancers are serving, and your dashboards are green. The runbook says to fail over, so someone starts working through it, and every step stalls: the scale-up call is throttled, the DNS update is accepted and does not take effect, the replica promotion times out twice and then succeeds forty minutes later.

Afterwards the incident is described as the cloud provider being down while our instances were fine. Both halves of that are true, and they are not in tension. The instances were fine because they were already running. The plan failed because every step in it was a request to change something, and the machinery for changing things was the part that was impaired.

Two planes, two reliability profiles

The data plane is what carries your traffic: the instances already running, the load balancers already configured, the DNS already resolving. Its failure modes differ from those of management APIs; consult the specific service and operation.

The control plane is what changes things: launching an instance, modifying a security group, updating a record, promoting a replica. It is a distributed system with its own dependencies, and it can be impaired independently. Recovery demand can also encounter throttling or capacity shortages.

Providers document this distinction and most architecture diagrams ignore it.

What that means for a recovery plan

Read any runbook and count the steps that require an API call to succeed. "Scale up the standby fleet." "Update the DNS weight." "Promote the read replica." "Fail over to the other region." Each one is a control-plane operation, and each is being attempted by every other customer in that region at the same moment.

The result is the failure people describe as the cloud provider being down when their instances were fine. The instances were fine. The plan needed an API that was not.

Static stability

The principle is that the system holds its current state, and continues to work, without needing to make a change. Concretely:

• Pre-provision the failover capacity. If losing a zone means the remaining two must serve everything, run them at a level where they already can. Capacity that has to be launched during the event is capacity you may not get, both because the API is degraded and because everyone else wants the same instance type.
• Prefer health-check failover over reconfiguration. A load balancer that stops sending traffic to unhealthy targets needs no API call. A script that removes a target group does.
• Cache what you depend on. A service that fetches configuration or credentials from a provider API on every start will not start during the event. Cache configuration only where stale values are safe. Do not extend expired credentials or bypass revocation; plan credential refresh and the permitted failure behavior separately.
• Know which of your failovers are data-plane operations. Anycast and health-checked DNS behave differently from a record update. One is already running; the other is a change.

The uncomfortable arithmetic

Static stability costs money, and that is the whole trade. With three equally sized zones, surviving one lost zone at the design load requires total capacity of at least 1.5 times that load: 50 percent extra relative to demand, or one third of provisioned capacity spare. Allow more for uneven routing and latency limits. Autoscaler headroom is a workload-specific policy, not a fixed 10 percent.

The counter-argument is not that static stability is too expensive, because that is a number a business can decide about. It is that a design which provisions during recovery is not cheaper; it has paid a different price, in accepting an API dependency, a capacity dependency and an instance startup time at the exact moment all three are least available. Both options cost something, and the only unacceptable position is the common one, where the spare capacity was never bought and the dependency on acquiring it during the event was never written down.

The test

Ask of any resilience plan: which steps require a control-plane call, and what happens to each if that call fails or takes twenty minutes? A plan with no such steps is statically stable. A plan with three of them has three ways to not work on the day, and the honest version of the plan says so.

Self-check

Three zones each provide 100 units of usable capacity, so 300 in total. Work out the largest steady load that survives the loss of one zone with no scaling, express it as a per-zone utilisation figure, and then generalise: what does the same calculation give for four zones, and for ten? Then take this three-step runbook and score it against the test above. Step one, promote the standby database. Step two, raise the autoscaling group's desired count from 6 to 12. Step three, shift traffic by lowering the failed zone's weight in DNS. Work through both before reading on.

The capacity half first. Losing one zone leaves 200 units, so the largest load that survives without scaling is 200. Each zone must therefore run at 200 / 300 ≈ 66.7 percent of its capacity at peak, and the common mistake is to read the answer as 50 percent, which is what you get by thinking about halving rather than about what the survivors carry. Generalising, n zones surviving one loss need each zone at (n−1)/n of its capacity: 3/4 = 75 percent for four zones, and 9/10 = 90 percent for ten. So the cost of static stability falls sharply as the number of zones rises, which is the real argument for spreading across three zones rather than two, where the figure is 1/2 = 50 percent and you are paying for a whole idle zone.

The runbook scores badly, and usefully. All three steps are control-plane operations, so a plan described as a failover procedure has three independent ways to not work on the day. Step two is the most exposed, because it needs both the API and the actual instance capacity, and every other customer in that region is asking for the same instance type in the same minutes. Step one needs an API that is itself part of the impaired region. Step three is the subtle one: a DNS weight change is a control-plane write, and even when it succeeds it then waits out the record's TTL at every resolver, so a 300-second TTL means the change is not a mitigation for five minutes and is not fully effective for longer.

The statically stable version of the same intent uses none of them. Run the surviving zones at 66.7 percent so no scale-up is needed, let the load balancer's health checks stop sending traffic to the failed zone, which is a data-plane behaviour already running and requires no call from you, and configure the standby database to be promoted by a managed failover the provider drives. The plan shrinks to observing that it worked, which is the property worth having.

Next: failure domains sets out which boundary contains which failure, and size for a zone loss makes this arithmetic an exercise.

Sources: AWS static stability.

For a private cloud, OpenStack service boundaries maps Nova, Placement, Neutron and Cinder to the evidence needed when an instance request fails.