TL;DR: Give every subscription its own isolated delivery lane: events fan in once, jobs are signed and queued per endpoint, delivery is at-least-once with backoff stretched over days, and a dying endpoint gets paused instead of consuming shared workers. Event IDs plus HMAC signatures are what let receivers trust replays.
How to approach it
Open with the delivery guarantee, because it shapes the client contract: networks and receiver outages force at-least-once, so receivers dedup on an event ID and you stop pretending ordering exists across subscribers. Then isolate per subscriber; the rest of the design follows from that decision.
A strong answer
Pipeline.
Contract. Use retrying delivery with possible duplicates and a finite retry horizon; prolonged receiver failure can end in terminal failure, not guaranteed eventual delivery. Every event carries a unique ID and an incrementing sequence per subscription, so receivers detect gaps and dedup replays. Ordering across event types is explicitly not promised; receivers who need it use the sequence numbers.
Isolation is the whole game, and the arithmetic shows why it cannot be left to good behaviour. Suppose a shared pool of 500 delivery workers and one merchant receiving 20 events a second whose endpoint stops responding, holding each attempt for the full 60-second timeout. By Little's law that endpoint alone occupies 20 × 60 = 1,200 workers of demand against a pool of 500, so it does not degrade the pool, it consumes all of it. Healthy merchants receiving thousands of events a second now wait behind one dead endpoint, and every delivery in the system slows.
The same arithmetic sets the limit that fixes it. Capping that endpoint at 10 concurrent attempts bounds its occupancy at 10 workers, 2 percent of the pool, whatever it does with them. One merchant's endpoint hanging for 60 seconds must not occupy workers needed by ten thousand healthy merchants. So: per-endpoint concurrency limits, per-endpoint queues with bounded depth, and dispatch that pulls fairly across endpoints. A merchant whose failure rate crosses a threshold, say 95 percent of attempts over the last hundred, gets paused automatically, notified on their dashboard, and resumed by scheduled redelivery or a manual button. Without this, one blackholed endpoint quietly becomes a fleet-wide delivery slowdown, and the cause hides inside aggregate latency charts for days.
Retry schedule. Exponential with jitter and a long horizon: roughly 1 minute, 5 minutes, 30 minutes, 2 hours, then hourly out to about 72 hours before giving up and marking the event failed.
Count what that schedule costs and covers, because both matter. The four backoff steps plus roughly 72 − 3 ≈ 69 hourly attempts is about 73 attempts per event over three days, which is the budget one undelivered event consumes. Compare a naive 30-second retry over the same window: 72 × 60 / 0.5 = 8,640 attempts, more than a hundred times the load, all of it aimed at an endpoint that is already down. The schedule is not politeness; it is what makes a 72-hour horizon affordable at all.
What the horizon buys is the other half. A merchant outage lasting an afternoon is covered by the first few hours of hourly retries and resolves with no human involved on either side. Three days covers a bad weekend or a failed migration. Beyond that the assumption breaks anyway, since a receiver down for a week usually needs a deliberate replay of a date range rather than delivery of a single stale event.
Signing. Sign each attempt with a fresh timestamp plus the exact body using the endpoint secret. Keep the logical event ID stable so receivers deduplicate later legitimate redelivery. Receivers verify the signature and reject timestamps older than a few minutes, closing the replay window. Rotation runs dual secrets through an overlap window so merchants migrate without breakage. Publish your egress IP list for teams that firewall inbound webhooks.
Destination controls. Tenant URLs are untrusted. Allow approved HTTPS destinations, block loopback/private/link-local and metadata addresses, validate resolved addresses again at connection time and bind the connection to that validated address. Disable redirects or revalidate every hop; enforce egress policy against DNS rebinding. Limit response bytes/time and redact delivery logs.
Payload discipline. Small, versioned, typed payloads (payment.settled.v2); large objects referenced by ID with a fetch API rather than bloating every delivery. Idempotency keys ride along so a receiver that processes twice can no-op safely.
Operations. Per-endpoint success-rate SLOs, delivery logs retaining request and response snippets for merchant-facing debugging, manual redelivery, and replay of an event class for a date range from the durable event log. Queue depth per endpoint is the health signal that predicts trouble before customers call.
What interviewers probe next
"The merchant demands strict ordering." Serial per-endpoint delivery provides it and destroys throughput for slow endpoints; sell sequence numbers plus dedup instead, and explain the trade you refused.
"They return 200 but drop the event internally." Undetectable from your side; the mitigation is tooling, delivery logs plus one-click replay, and documentation telling receivers to persist before acknowledging.
"Rotate the signing secret without an outage." Accept old and new during the overlap window, show the pending secret in the dashboard, retire the old one after a grace period.
Common mistakes
Retrying every 30 seconds for hours: it DDoSes the recovering merchant and starves your own workers.
Signing bodies without a timestamp, leaving replay attacks wide open.
One shared queue for all endpoints, where a dead subscriber's backlog delays everyone and nobody can tell why.
References