TL;DR: For checkout diagnostics, send the required trace stream to a trace-ID-aware tail-sampling tier and retain errors, slow traces and a baseline sample within a measured capacity budget. If you head-sample first, explicitly accept missing error traces. No tail policy can recover spans discarded upstream.
How to approach it
Ask for request volume, average spans per trace, and what decisions depend on traces. Volume picks the strategy more than any architecture preference does; saying that first signals you have paid a tracing bill before.
A strong answer
Head sampling decides at span creation. A parent-based ratio sampler propagates a consistent decision when participating SDKs honor the parent context. It is cheap, but instrumentation and exporting sampled spans still cost resources. The initial decision cannot know a future error or completed duration. At 5 percent sampling, an incident affecting 0.1 percent of requests produces an expected request_count × 0.001 × 0.05 retained affected traces. Whether that is adequate depends on volume and the required evidence; rare failures can be missed entirely.
Tail sampling decides after the fact. A gateway collector buffers arriving spans for a decision window, then applies policies: keep on error attribute, keep when end-to-end latency crosses a threshold, keep rare classes (new deploy version, unusual route, flagged test user). It preserves outliers at meaningful rates while discarding boring healthy traffic. Costs are real. At 2,000 traces per second, 40 spans per trace and a 60-second decision window, the tier may retain roughly 4.8 million spans before accounting for early decisions, skew and overhead. Divide by replicas only after validating distribution and per-trace size. Buffering and export hops delay telemetry availability; asynchronous SDK export avoids putting that entire delay on checkout. Route each trace to one decision point, for example with the OTel load-balancing exporter. A generic connection-based balancer can scatter a trace assembled by multiple services; it is not a substitute for trace affinity.
For checkout, start from the evidence requirement. If the aim is to retain all observable error traces, record/export that stream without a probabilistic head filter and size the collector for the full arriving volume. Apply error and latency policies there, with a baseline sample of successful traffic. Collector failures, late spans and transport loss still limit completeness. If full collection exceeds the budget, lower the upstream ratio and document the resulting loss instead of promising that tail sampling restores it.
Define what happens when an error storm exceeds the budget: increase capacity, narrow the retained payload, or shed with a visible loss counter. An absolute ingest cap and unconditional error retention cannot both hold at arbitrary error volume. At low measured volume, retaining all traces with short retention may be simpler than a tail tier. Payment audit records need their own durable controls; traces alone are not that record. OpenTelemetry sampling.
| Strategy | Decides | Cost shape | Blind spot |
|---|---|---|---|
| Head ratio | SDK at trace start | Low decision cost; sampled export still costs | Future errors and slow traces can be missed |
| Tail policies | Gateway after decision window | Buffered trace data and routing state | Upstream loss, late spans, policy and capacity limits |
| Always-on | Record/export every span | Full collection and retention cost | Transport or collector failures still lose data |
One discipline makes the hybrid auditable: compare retained error traces with independent request/error counters and known synthetic cases. Record the difference and partial-trace rate; the trace backend alone cannot count traces it never received.
What interviewers probe next
"Where does trace-ID affinity break in practice?" Different services can send spans of one trace over different connections; ordinary connection balancing can route those to different gateways. Membership changes can also move a trace during its decision window. Inject known multi-service traces and verify completeness during scaling and rollout.
"Does buffering delay incident evidence?" The decision window, queueing and export retries determine delay; there is no universal seconds-only bound. Page from independent request metrics and expose trace-pipeline lag so investigators know when evidence is incomplete.
"How do you prove sampling hid nothing?" You cannot prove universal completeness from retained traces alone. Test known failure cases against independent counts, exercise overload and late spans, then state the measured loss bounds and untested cases.
Common mistakes
Answering "sample one percent uniformly" as if the question were about storage arithmetic, without measuring how many rare failures that ratio is likely to retain.
Enabling tail policies without trace-ID-aware balancing, shipping partial traces nobody notices until the postmortem that needed them.
Keeping everything forever because ingest seemed cheap, then paying the query tax when p99 hunts scan billions of spans.