Idempotency is a design property, not a retry setting

TL;DR: Exactly-once claims need a boundary. A database transaction can atomically record a key and its effect. An external charge needs the provider’s idempotency contract or reconciliation of an uncertain outcome; writing an intent locally does not guarantee that the charge happens exactly once.

The situation that forces it

A client sends a request. The server processes it, commits, and the response is lost, or the connection drops, or the client times out one millisecond early.

From the client's side those are indistinguishable from the server never having received anything. The client holds exactly one piece of evidence, which is the absence of a response, and that absence is produced equally by a request that never arrived and by a request that succeeded and whose answer was lost.

So the client retries, because not retrying means silently dropping work that may have failed, and the server now sees the same request twice. Notice which end of the connection can fix this. The client cannot: no timeout value, no retry policy and no amount of care distinguishes the two worlds in that diagram. Only the receiver knows whether it already did the work, which is why tolerating duplicates is a property of the receiver rather than a setting on the caller.

Acknowledgements can be lost too, in both directions. Systems can provide exactly-once processing within a defined transactional boundary, but that guarantee does not automatically extend across an external API call. So the practical question on any given handler is which state changes commit together, and how retries are deduplicated outside that boundary.

Three mechanisms, in order of preference

A natural key with a uniqueness constraint. If the operation creates a row that has an identifier the caller already knows, let the database enforce it. INSERT ... ON CONFLICT DO NOTHING against a unique index makes the second attempt harmless with no extra machinery. This is the best option available and it is the one people skip because it feels too simple.

A caller-supplied idempotency key. Where the operation has no natural key, the caller generates one per logical operation and sends it with every attempt. The server stores the key with the result of the first execution, and any later request carrying the same key returns the stored result rather than executing again. Two details make or break it: the key must be stored in the same transaction as the effect, or a crash between them recreates the problem you were solving; and the deduplication retention window must be documented, because a retry after the key is removed may execute again. Bind the key to the caller and original operation parameters so it cannot silently refer to a different request.

A ledger, written before acting. For an external effect, record the intent durably and assign a stable operation ID before calling the provider. Pass that ID through the provider’s idempotency mechanism when available. If a charge succeeds but your worker crashes before recording completion, the local ledger cannot distinguish success from failure. Reconcile against the provider’s status API or retry with the same supported key, within its retention window; otherwise leave the operation unresolved for investigation rather than issue a blind second charge. Stripe’s idempotency contract is one concrete example: it stores the first executed result and may prune keys after at least 24 hours.

What duplicate delivery costs when nothing absorbs it

The consequence is easier to take seriously with a number on it, so take one. These inputs are illustrative and chosen to be easy to follow; substitute your own.

A payments service handles 50,000 charges a day. Suppose 2 percent of calls to the provider time out from the client's point of view, which is 50,000 × 0.02 = 1,000 uncertain outcomes a day. Suppose further that most of those timeouts are late responses rather than lost requests, so 90 percent of them had already committed at the provider: 1,000 × 0.9 = 900 charges a day that succeeded and were reported to the caller as failures.

Now apply a retry policy of two further attempts. With no deduplication at the receiver, each of those 900 becomes up to three charges instead of one, so the arithmetic is 900 × 2 = 1,800 duplicate charges a day. At an average charge of 40 currency units that is 1,800 × 40 = 72,000 a day being taken from customers who bought one thing, and the number is generated entirely by a retry policy that is doing exactly what it was configured to do.

Two things this calculation does not establish. It does not tell you your own timeout rate or the split between lost requests and lost responses, both of which have to be measured rather than assumed, and it says nothing about the refund and support cost, which in practice exceeds the charge value. What it does establish is the shape of the risk: the exposure scales with traffic and with retry attempts, so it grows precisely as the system succeeds.

What idempotency is not

It is not a retry policy. Retries are the pressure; idempotency is what makes the pressure safe. Configuring retries without a receiver that tolerates them converts a transient failure into a duplicate charge.

It is also not the same as a request that happens to be a read. Reads are naturally idempotent, which is why the interesting cases are all writes, and why the mechanisms above all concern themselves with storing something.

The cases that break it

Non-deterministic values. A retry that generates a fresh identifier and creates another row duplicates the effect. A different response timestamp alone does not make an operation non-idempotent: idempotency concerns the intended effect on server state. Preserve operation identity across retries and store the result when the API promises response replay. HTTP semantics defines this distinction.

Partial effects. An operation writing three rows and calling two services is idempotent only if every step is, or if all of it is inside one transaction. The half that is not is the half that bites.

Ordering. Idempotent does not mean commutative. A retry that arrives after a later update can overwrite it, which is why version numbers or conditional updates belong alongside the key.

The habit that follows

Ask of any handler: what happens if this runs twice? Ask it during design rather than during the incident. In a system built from queues, retries and timeouts, running twice is not an edge case, it is Tuesday.

Self-check

A worker consumes an order queue. For each message it inserts a row into shipments, calls a carrier API to book a collection, and writes the returned tracking number back to the row. The queue delivers at least once, so some messages arrive twice. Decide which of the three mechanisms applies to each of the three steps, and say what happens today if the worker crashes after the carrier call and before the write. Work it through before reading on.

The insert is the easy one, because an order has an identifier the caller already knows. A unique index on order_id with ON CONFLICT DO NOTHING makes the second delivery harmless with no extra machinery, which is the first mechanism and the one worth checking for before reaching for anything else.

The carrier call is the hard one, because the effect is outside your database and no constraint of yours can reach it. This is the third mechanism: assign a stable operation identifier when the shipment row is created, pass it through the carrier's own idempotency mechanism if they offer one, and treat their API as the authority on whether a collection is booked.

The write-back is the second mechanism, and it is also where the described design is already broken. If the worker crashes after the carrier books the collection and before the tracking number is stored, your database says no collection was booked and the carrier says one was. The redelivered message then books a second collection, because nothing in your own state records that the first call happened. The fix is not a longer timeout or fewer retries; it is that the operation identifier has to be durable before the external call, so the retry has something to present that the carrier can recognise as the same request.

Then the question the exercise is really asking: a local intent record cannot prove that no charge or booking happened, because it only ever records what you meant to do. Resolve the uncertainty against the provider, by reconciling on the operation identifier or retrying with the same supported key inside its retention window. Issuing a fresh logical payment because a response was lost is how the 1,800 duplicates above get created.

Next: the dead-letter queue question works through what to do with the messages that fail every attempt, and active-active write conflicts is what happens when two regions each apply an operation that was only supposed to happen once.

For a key-value data model, NoSQL access patterns and consistency shows why an eventually consistent lookup cannot decide whether a unique operation may proceed.

The queue acknowledgment boundary explains when completed work can be delivered again. A transactional outbox and inbox connects local database effects to durable publication and consumption.

REST API contracts adds conditional updates and conflict responses so a repeated request cannot silently overwrite a later edit.