TL;DR: Clients send a unique idempotency key per logical operation; the server durably claims the key, and atomically records effects only where they share its database transaction, replays the stored response on retries, and rejects concurrent duplicates until the first finishes. External payment effects require provider idempotency and reconciliation of uncertain outcomes.
How to approach it
Open with why this is unavoidable: a timeout is indistinguishable from a lost-after-commit response, so networks force at-least-once delivery onto callers, making deduplication the receiver's job. Then design the key lifecycle: creation, storage, replay, expiry.
A strong answer
The core mechanism. Every mutating request carries an Idempotency-Key header, generated client-side (a UUID per logical attempt, not per retry). The server's flow:
The critical property is atomicity between doing the work and recording the key. For a local ledger operation, one database transaction can insert the key, update ledger rows and store the response. An external card provider cannot join that ordinary transaction: persist a pending intent/outbox, call with a stable provider key, and reconcile ambiguous timeouts before marking completion. Local database failure rolls back uncommitted local effects. A process crash after an external provider accepts a charge leaves an uncertain state; retry/query using the same provider reference, never assume rollback undid the charge. If two identical requests race, the primary-key constraint on the key row lets exactly one proceed while the other gets a conflict it must treat as "in flight", not as failure-to-retry.
What goes wrong without each piece:
- Key checked outside the write transaction: two concurrent requests both see "not seen" and both charge.
- Response not stored: the retry gets a fresh success response for a new execution... unless you also made execution conditional, which is harder to prove than storing the outcome.
- Key scoped to one pod or cache only: failover to another node loses dedup state right when timeouts are likeliest. Keys belong in durable shared storage next to the business data, not in Redis-with-TTL alone.
API contract details that separate practitioners from tutorial readers:
- Scope keys to the operation and principal: namespace keys by authenticated principal and operation. Within that scope, store a canonical payload hash and reject key reuse with different parameters; silent acceptance means a buggy client reusing UUIDs charges the wrong person once instead of twice.
- Store enough to replay faithfully: status code plus body, so the retried client receives byte-equivalent success rather than a confusing second confirmation format.
- Expire keys honestly: Stripe-style 24-hour retention balances dedup usefulness against storage; document it, because a client retrying after expiry will genuinely double-charge unless amounts are also guarded by business-level uniqueness (a unique constraint on order ID is the usual second line).
- Make retries safe by default elsewhere too: GETs are naturally safe; webhooks outbound need their own dedup on the receiver side; batch endpoints need per-item keys or a deterministic item ordering.
Where exactly-once actually lands. The transport stays at-least-once; what becomes exactly-once is the effect, because outcomes are recorded under keys that make every duplicate a read. Downstream systems need the same discipline: if charging ultimately hits a card network, your internal key travels as their idempotency reference, subject to their idempotency scope, retention and status-query contract. A provider without these controls requires reconciliation/manual resolution; local recording alone cannot guarantee one external effect.
What interviewers probe next
"Client generates two keys for one logical payment (buggy retry logic). Now what?" Idempotency keys dedupe retries, not mistakes. The second line of defence is business uniqueness: one order maps to one successful charge, enforced by schema constraints. Both layers ship together.
"Why not just make operations naturally idempotent (set balance = X)?" Works where semantics allow; payments rarely do (fees, ledgers, notifications all append). Key-based dedup generalises where state-overwrite semantics cannot exist.
"Storage grows forever with keys?" Expire after a retention window aligned to realistic retry horizons, archive to cold storage for audit, and keep the unique constraint window explicit in documentation.
Common mistakes
Generating the key server-side on first contact. The client must own it: only the client knows which attempts are retries of one operation versus genuinely new ones.
Treating the idempotency layer as optional hardening. For money movement it is part of the API contract on day one; retrofitting it after partners integrate is brutal.
Testing only the happy path. The tests that matter: kill the server mid-transaction and retry, fire two concurrent identical requests, replay after commit. All three must produce one charge.
References