DevOpsInterviews logo
← Cloud Platforms & Architecture / 15
mediumAmazon & AWSRazorpayJPMorgan Chase

The database failed over to its Multi-AZ standby and the application fell over anyway. What did Multi-AZ promise, what does it not?

Database redundancy still needs reconnecting clients. Distinguish the deployment type, then measure how promotion and DNS changes affect application requests.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: A traditional RDS Multi-AZ DB instance synchronously replicates to a non-readable standby and automatically fails over. Typical failover is 60–120 seconds, with workload-dependent variation, not a guaranteed outage ceiling. Applications fall over during failover when they treat the DB endpoint as permanently reachable instead of retrying through a brief outage.

How to approach it

Separate the two promises cleanly: data safety versus service continuity. Then explain why a correctly functioning Multi-AZ setup can still take your application down for a minute or two, because that paradox is the whole question.

A strong answer

What a traditional Multi-AZ DB instance does. The primary instance replicates synchronously to a standby in another AZ. On primary failure (host loss, AZ impairment, maintenance), RDS promotes the standby and moves the DNS endpoint to it. Synchronous replication means committed transactions are on both copies, so failover loses nothing acknowledged. That is a durability and recovery-time feature.

What it does not do. Failover takes time. DNS must update, and clients may cache the old address or retain old connections; the standby must promote, and clients holding pooled connections find every socket dead. AWS describes 60–120 seconds as typical for this deployment, with larger transactions or recovery work able to extend it. The separate Multi-AZ DB cluster product has two readable standbys and different failover behavior. Identify the product before promising timing or read capacity. DB-instance failover and Multi-AZ deployment types.

So why did the application "fall over" rather than degrade? Because most applications treat connection failure as fatal:

  • Pools abort on first error. A connection pool that marks itself broken on one reset requires restarts or long timeouts to recover. Pools should validate-and-replace connections lazily after errors.
  • No retry budget on transient failures. One failed query during promotion should be retried with backoff if it was read-only or idempotent; instead the request 500s, upstream retries amplify, and the app converts a 60-second database blip into a much longer error storm. The next section puts numbers on how much longer.
  • Long-lived TCP assumptions. Services that open a connection at boot and assume it forever die at promotion and stay dead until redeployed.
  • DNS caching. JVM DNS caches, custom resolvers and sidecars pinning the old IP extend the window well past the provider's own.

Worked example: how a 90-second failover becomes a 13-minute incident

The amplification is the part worth computing, because it explains why the outage outlasts the database event. Illustrative figures: the service takes 1,000 requests per second, the database serves up to 2,000 per second when healthy, and failover takes 90 seconds.

During the failover, 1,000 × 90 = 90,000 requests fail. Suppose each is retried up to three times with backoff, which is ordinary and correct behaviour. That is up to 90,000 × 3 = 270,000 attempts queued against a database that is still promoting.

Now the recovery. The database comes back with 2,000 per second of capacity while normal traffic continues to consume 1,000, leaving 2,000 − 1,000 = 1,000 per second of spare capacity to absorb the backlog. Draining 270,000 attempts at that rate takes 270,000 / 1,000 = 270 seconds, about four and a half minutes of elevated latency after the database was already healthy.

Add a second retry layer, which is easy to have without deciding to: an upstream service or API gateway that also retries three times turns each original request into 3 × 3 = 9 attempts, so 90,000 × 9 = 810,000 and 810,000 / 1,000 = 810 seconds, about thirteen and a half minutes. Multipliers compose across layers rather than adding, which is why nobody predicts the total from their own layer's configuration.

Three things this establishes and one it does not. It establishes that the tail of the incident is produced by the retry policy rather than by the database, that the fix therefore lives in the clients, and that retry budgets matter more than retry counts, since a budget capping retries as a fraction of live traffic bounds the backlog no matter how many layers exist. What it does not establish is your own numbers: the spare capacity figure is the one that decides the answer and it is usually smaller than assumed, because a freshly promoted standby has cold caches and serves below its steady-state rate exactly when the backlog arrives.

Jitter matters here too, and it is invisible in this arithmetic. Retries without randomised backoff synchronise into waves, so the same 270,000 attempts arrive in spikes that exceed the spare capacity rather than spreading evenly across it.

The fix list is short and worth stating as commitments: idempotent writes where possible so retries are safe, aggressive-but-bounded retry with exponential backoff and jitter around database errors, connection pools configured with sane validation and reconnect behaviour, and health checks that mark the service degraded rather than dead while the database is briefly gone. Then rehearse: trigger failover in staging monthly (aws rds reboot-db-instance --db-instance-identifier STAGING_DB_ID --force-failover) and measure what the application actually experiences, because the gap between belief and reality here is always surprising.

So the accurate way to describe what you bought: Multi-AZ automates promotion and guarantees no acknowledged write is lost, which is a durability property. It does not put an upper bound on application outage, and as the arithmetic above shows, the application's own retry behaviour can make the outage several times longer than the failover. Whether those tens of seconds are invisible to users is decided by client-side engineering that the database product cannot do for you.

The database failed over correctly and the application fell over anyway WHAT MULTI-AZ PROMISED WHAT THE CLIENT DID 1 The primary fails host loss, or maintenance 2 The standby is promoted it already had every commit 3 The endpoint moves and DNS has to propagate 4 No acknowledged data lost that part is delivered 5 Every pooled socket is dead in every instance at once 6 The pool marks itself broken on the first error it sees 7 Requests fail, not retry including idempotent reads 8 Upstream retries amplify sixty seconds becomes ten minutes Synchronous replication means committed transactions are on both copies, so failover loses nothing acknowledged. That is a durability and recovery-time feature, and it is not an availability guarantee for your application. Clients may also cache the old address or hold old connections, so the promotion being quick does not make the client's recovery quick. The three client assumptions that convert this into an outage: a pool that aborts rather than replacing connections lazily, no retry budget on a transient failure, and a service that opens a connection at boot and assumes it forever.

What interviewers probe next

"Why not just use Aurora?" Aurora's storage layer replicates at six endpoints across three AZs and fails over faster, but the client-side truths are identical: connections break and retries decide whether users notice.

"Read replicas versus the standby?" Different purposes: replicas scale reads (and lag, which introduces staleness bugs); the traditional DB-instance standby supports failover rather than reads. Multi-AZ DB cluster standbys can serve reads. Confusing them in design reviews is common.

"How would you prove the fix works?" Scheduled failover drills with dashboards measuring user-visible error rate, not just database metrics. If nobody saw anything, you passed.

Common mistakes

Answering that Multi-AZ means high availability, full stop. It reduces downtime; it does not eliminate it, and the difference lands on the application.

Blaming the provider because failover took ninety seconds. That can be within typical observed timing. Compare the actual outage with the documented product behavior and application recovery objective rather than treating typical timing as a contract.

Never drilling failover before production needs it. The first real promotion is the worst possible time to learn how your pools behave.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.