TL;DR: The line ships to ingest nodes, pipelines parse it, routing hashes it to a primary shard where it becomes searchable after refresh while the translog guards durability, segments merge in background, and ILM rolls indexes through hot, warm, cold and delete. Ingest volume, query load, shard-count overhead and shard size all affect capacity.
How to approach it
Narrate the path in order and attach one failure mode to each stage. The interviewer is checking whether you know the difference between visibility (refresh) and durability (flush), and whether you have ever had to explain a yellow cluster at 3am.
A strong answer
Ship and parse. An agent (Filebeat, Fluent Bit, OTel collector) batches lines with disk-buffered backpressure and ships to ingest nodes, where ingest pipelines apply processors: grok or dissect for parsing, date for timestamps, geo or user-agent enrichment if wanted. Parsing costs CPU, which is why structured JSON at source wins: the pipeline becomes field validation instead of regex archaeology, and failed documents stop landing in dead-letter queues.
Route and write. The document's _id hashes modulo the primary shard count to pick its shard. That count is fixed at index creation; changing it later means reindex or split, which is why index templates deserve design attention. On the shard, the document enters the memory buffer and appends to the translog. With request durability every write fsyncs before acknowledgement; async buys throughput at the cost of a few seconds of loss window. Keep request durability unless the log owner accepts the documented loss window; audit and security records may require stronger retention guarantees.
Refresh makes it visible. Roughly every second the memory buffer flushes into a new Lucene segment, and the segment becomes searchable. This is the near-real-time gap: a log line can exist durably yet be invisible to Kibana for up to the refresh interval, and teams cranking refresh intervals down during load spikes discover that visibility has a throughput price. Segments are immutable; deletions are tombstones until background merges reclaim them.
Why the rollover size is the number to get right
A 50 GB rollover looks arbitrary and is doing two jobs, both of which are arithmetic.
It sets shard count, and shards are not free. A cluster ingesting 1 TB a day at 50 GB per shard creates 1,000 / 50 = 20 primary shards a day, and holding 14 days hot means 20 × 14 = 280 primaries, doubled to 560 with one replica. Each shard carries heap and file-handle overhead on the node that holds it, so the practical guidance of keeping shards in the tens of gigabytes is what stops a cluster from dying of shard count rather than of data volume.
It also sets recovery time. A lost node has to have its shards rebuilt or relocated, and that work is proportional to shard size: 50 GB shards move in minutes, 500 GB shards move in hours, during which the cluster runs without a replica. Pick the number from those two constraints rather than from a template, and note which direction each pulls, since smaller shards mean faster recovery and more of them to keep track of.
ILM moves old data somewhere cheaper.
| Phase | Trigger | Actions | Typical hardware |
|---|---|---|---|
| Hot | rollover at 50 GB or 1 day | indexing, full replicas | NVMe, most RAM |
| Warm | age 7 days | read-only, force-merge to 1 segment, trim replicas | SATA SSD |
| Cold | age 30 days | searchable cold or snapshot mount | object store, HDD |
| Delete | age 90 days | remove index | n/a |
Set ages from measured query depth, not compliance vibes: check how far back anyone actually searched, keep the mandated event classes on their own stream, and let the rest roll cheaply. OpenSearch speaks ISM policies instead of ILM; same ideas, different wiring, and AWS interviewers notice whether you know that.
Where clusters actually die. Too many small shards: every shard is a Lucene instance with heap overhead, so thousands of tiny hourly indexes exhaust heap long before disk fills; aim for shards in the tens of gigabytes and bound total shards per node (cluster.max_shards_per_node). Oversized shards slow recovery and rebalance and stretch merge GC pauses. Mapping explosions from dynamic fields in semi-structured logs bloat cluster state; set explicit mappings or disable unknown fields. And watermarks: default percentage watermarks commonly include 85 percent low and 95 percent flood stage, with max-headroom and version settings affecting the threshold. Low watermark restricts allocation but allows some newly created primary shards; flood stage applies a read-only-allow-delete block to affected indices with shards on the pressured node, which is why "Kibana shows nothing" sometimes means "the disk filled overnight".
What interviewers probe next
"New logs stopped appearing. Where do you look?" Shipper backlog, ingest pipeline failures, allocation blocked by watermarks, refresh interval cranked up under load, in that order; each leaves a distinct fingerprint in APIs you can curl in minutes.
"Cluster turned yellow overnight." Replicas unassigned: node loss, watermark pressure, or too few nodes for replica count. All primaries are assigned in yellow health, but reduced redundancy remains a risk; searches may still serve, and the fix is capacity math rather than panic reindexing.
"Template changed but old data looks wrong." Templates bind at rollover, so corrections apply only to future indexes; correcting historical mappings or values generally needs a reindex into a corrected destination. Splitting increases primary shard count and does not repair malformed values or field types.
Common mistakes
Explaining "Elasticsearch is like a database" and stopping there, revealing no operational scar tissue.
One-replica prod log indexes on shared disks, then genuine surprise when a node loss takes evidence with it.
Confusing refresh with flush in follow-ups, claiming data is "lost" when it is merely not yet visible.
References