DevOpsInterviews logo
← DevOps System Design & Architecture / 13
mediumGoogleSnowflakeJPMorgan Chase

A network partition split your database cluster in half and both sides kept serving writes. What does CAP predict about this?

CAP is not a menu of two letters per system. The scored answer knows what the theorem claims (narrowly), what most engineers misquote, and what a split-brain costs when both halves accept writes.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: CAP says a partitioned distributed system must sacrifice either consistency or availability for the duration of the partition. Both halves accepting writes means someone chose availability and got split-brain: conflicting data that reconciliation must untangle. Quorum systems choose consistency by refusing the minority side.

How to approach it

State the theorem precisely and narrow first: it is about partitions specifically, it is binary only during the partition, and "choosing" happens per operation via timeouts. Then explain what each half of a partitioned cluster does under the common configurations, because that concrete behaviour is what interviewers want.

A strong answer

The claim, stated honestly. In a distributed system supporting reads and writes, when the network partitions, you cannot simultaneously guarantee linearisable reads/writes (consistency) and successful responses from every reachable node (availability). You will sacrifice one until healing. That is all: nothing about latency, nothing about backups, nothing about "pick 2 of 3" as a permanent identity.

What your scenario actually did. Both halves serving writes is the classic split-brain. Under leader-based replication with no quorum requirement, each partition elects its own leader (or keeps operating on stale leadership information) and accepts commits the other side cannot see. When the partition heals, the cluster holds divergent histories: same rows written differently on each side, possibly unique keys allocated twice.

The standard defences, and what each sacrifices:

ConfigurationPartition behaviourWhat it protects
Majority write quorum plus defined version/order rulesA write needs more than N/2 replicas; read semantics still matterIntersecting write sets, not automatic linearizability
Leader lease + clock-bounded authorityOld leader stops serving after lease expiry even if partitionedSingle writer
Manual/automatic reconcile of both-write historyAccepts divergence, merges laterAvailability
Consensus (Raft/Paxos)Majority continues; minority unavailableCorrectness + automatic recovery

Consensus systems are the industry default precisely because they make the choice mechanical: a majority partition can continue committing when leadership and required dependencies function; the minority returns unavailability errors, which CAP calls sacrificing availability, but only for the minority slice and only while partitioned. The "C or A" framing feels dramatic; real deployments experience it as "the app in the small datacentre gets errors until the link heals".

Quorum is where the abstraction becomes arithmetic, and the numbers explain several conventions that otherwise look arbitrary. A majority of N is floor(N/2) + 1, so 3 nodes need 2 and tolerate 1 failure, while 5 need 3 and tolerate 2. Note what that buys per node added: going from 3 to 5 costs two more machines and raises fault tolerance from 1 to 2, whereas going from 3 to 4 needs floor(4/2) + 1 = 3, still tolerating only 1. An even-sized cluster pays for a node that adds no tolerance and one more thing that can fail, which is why cluster sizes are conventionally odd.

The two-node case is the one worth stating plainly: a majority of 2 is 2, so a partition leaves neither side with a quorum and both stop accepting writes. That is not a misconfiguration, it is the arithmetic, and it is why a pair of nodes cannot be made highly available by adding replication alone. A witness or arbiter exists precisely to make the count odd without paying for a full third replica.

The part that gets missed most often: the choice is per-operation, made implicitly by timeout values. Retrying an unavailable minority operation for ten minutes does not restore CAP availability during an unbounded partition. Timeouts change client waiting and uncertainty; the replication/coordination protocol determines which operations may succeed. R+W>N gives read/write intersection but alone does not impose a majority write quorum or resolve concurrent writes.

Reconciliation, if divergence happened anyway. Last-writer-wins by timestamp needs synchronized clocks and still silently drops one write. Version vectors preserve both writes for application-level merge (shopping carts merge by union; bank balances cannot). The honest engineering position: prevent dual-writes with consensus or fencing tokens rather than reconciling afterwards, because semantic merges push business rules down into the infrastructure layer, and that is where money bugs live.

What interviewers probe next

"Where does PACELC add value?" It names the trade-off everyone actually tunes: even without partitions, latency versus consistency persists (write to quorum = slower but safer; async replica = fast but stale). CAP covers the emergency; PACELC covers Tuesday.

"Is DNS CP or AP?" Trick question worth enjoying: ultimately consistent for updates (AP-ish) but with TTLs everywhere, so the interesting properties live in caching layers, which is why CAP labels fit systems poorly once caches exist.

"How do you detect split-brain quickly?" Fencing tokens and epoch numbers: every write carries a term/epoch, storage refuses older epochs, and monitoring alarms on multiple active leaders. Detection must be built-in, not discovered by accounting discrepancies.

Common mistakes

Reciting "you pick two of three" with databases labelled C, A or P. Partitions are events, not configurations, and real systems choose differently per request path.

Assuming async replication plus failover gives consistency. Failover of an async primary can lose acknowledged writes; that is durability loss without any partition drama.

Treating reconciliation as an implementation detail. If both-side writes are possible in your design, deciding merge semantics IS the design.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.