TL;DR: For a payments API with enough representative traffic, choose a canary with explicit promotion and abort conditions. Reserve capacity for recovery, compare equivalent request cohorts, and check that the old version can read everything the new version writes. Routing traffic back cannot undo a payment or repair corrupted data.
How to approach it
State your assumptions before choosing a strategy. A stateless HTTP service with weighted routing has different options from a worker consuming a shared queue. Ask about request volume, duplicate-payment protection, spare capacity and database changes. Then explain how your preferred rollout would fail and how you would detect that failure.
A strong answer
Rolling replaces replicas in batches. Kubernetes maxSurge permits extra replicas; maxUnavailable permits a reduction in available replicas. If the cluster cannot schedule the surge and no unavailability is allowed, progress can stall. Without weighted routing, the proportion of new replicas only approximates their traffic share. Long-lived connections and uneven load can skew it. Returning to the old version requires another rollout, with image pulls, scheduling and readiness checks.
Blue-green keeps two application environments and switches traffic between them. The second fleet needs capacity during the overlap, although shared infrastructure means the total bill need not double. Switching back is quick only if the old fleet remains healthy and the router, connection draining and application state allow it. Both versions often share a database. Separate databases introduce replication and reconciliation work instead of removing the state problem.
A canary exposes a selected cohort to the candidate version while stable continues serving the rest. Weighted routing lets traffic share differ from replica share, which helps when a small cohort still needs multiple replicas for availability. The stable fleet must retain enough capacity to receive traffic on abort. Argo Rollouts describes how traffic weights and replica scaling interact.
| Strategy | Capacity to plan | Recovery mechanism | Important limitation |
|---|---|---|---|
| Rolling | Surge replicas and terminating workload overlap | Roll out the previous artifact | Recovery needs scheduler capacity and compatible state |
| Blue-green | Two ready fleets during overlap | Switch routing to the retained fleet | A fast switch does not reverse writes |
| Canary | Candidate replicas plus stable recovery capacity | Stop promotion and return traffic to stable | Shared dependencies can spread damage beyond the cohort |
For this API, I would use canary after verifying payment idempotency and schema compatibility. I would compare candidate and stable by endpoint and relevant payment-provider cohort, recording request counts alongside errors and latency. A business check would look for duplicate authorisations or a mismatch between accepted requests and durable payment records. An HTTP 200 alone cannot establish correctness.
Worked example: a green dashboard with little evidence
Consider a fictional API receiving 200 requests per second. A 1% canary receives about two requests per second, or 600 requests in five minutes. At a 0.1% background failure rate, the expected count is only 0.6 failures in that window. Seeing zero failures is quite plausible even when the candidate has a meaningful regression. Retries and correlated failures also make a simple independent-sample interpretation unreliable.
Suppose a rarely used provider carries 0.5% of requests. Random allocation gives the canary only about three requests from that provider in the same window. An overall p99 graph can look healthy while telling us almost nothing about that path. I would hold promotion until the relevant cohorts have adequate observations, and supplement production observation with provider-specific checks that avoid unintended financial transactions.
Write down the policy before rollout: which observations are required, which failures trigger abort, and what happens when telemetry is missing. Missing data should hold promotion and alert the owner under this policy. It should not silently pass. Argo analysis supports successful, failed and inconclusive outcomes; configure those outcomes deliberately and rehearse a metrics-provider failure.
A longer soak helps reveal connection leaks and resource accumulation, but elapsed time alone is insufficient. Record traffic volume, memory trend and pool utilisation. A quiet overnight hour may exercise less of the system than five minutes at peak load.
What interviewers probe next
"The canary fails at 5%. What happens?" Stop promotion and return traffic to stable if that is the rehearsed recovery action. Verify stable capacity, queue age and payment reconciliation after the switch. If shared state is damaged, routing recovery and data recovery become separate incident tasks. Preserve the candidate's logs and version identifiers for diagnosis.
"Can you canary a database change?" Some changes can be scoped to a tenant or shard, but application traffic percentages do not constrain a shared DDL operation. Sequence compatible schema and code changes, measure lock and replication effects, and keep old representations until rollback no longer needs them. Dual writes require failure handling and reconciliation; they are not automatically required for every migration.
"What if there is almost no traffic?" Use representative tests and an explicitly observed release with a retained recovery option. Blue-green can make the routing switch convenient, but synthetic success still leaves production behaviour uncertain. State what remains untested before increasing exposure.
Common mistakes
Using a fleet-wide error rate that hides a candidate regression. Compare version-specific cohorts and also watch shared dependency health.
Assuming 1% traffic means at most 1% impact. A bad query can exhaust the shared database pool, and a faulty writer can change data used by stable.
Calling an old image a complete rollback plan. Review the artifact and its surrounding configuration, then test recovery against the state produced by the candidate.